August 2018 - Infra - oVirt List Archives

[CQ]: 93918, 7 (ovirt-provider-ovn) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 29 Aug '18

29 Aug '18

A system test invoked by the "ovirt-master" change queue including change 93918,7 (ovirt-provider-ovn) failed. However, this change seems not to be the root cause for this failure. Change 93982,3 (ovirt-provider-ovn) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 93982,3 (ovirt-provider-ovn) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/93918/7 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/93982/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9220/

1 0

[CQ]: 93982, 3 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 29 Aug '18

29 Aug '18

Change 93982,3 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93982/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9217/

1 0

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1810
by jenkins＠jenkins.phx.ovirt.org 29 Aug '18

29 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1810/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d Commit message: "adding kubevirt/containerized-data-importer to stdci v2" > git rev-list --no-walk 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins8049150100185792854.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/packages error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/packages' Build step 'Execute shell' marked build as failure

1 2

[JIRA] (OVIRT-2457) mirrors.phx running out of RAM/swap
by Evgheni Dereveanchin (oVirt JIRA) 29 Aug '18

29 Aug '18

Evgheni Dereveanchin created OVIRT-2457: ------------------------------------------- Summary: mirrors.phx running out of RAM/swap Key: OVIRT-2457 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2457 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Evgheni Dereveanchin Assignee: infra Since we added monitoring it became obvious that the mirrors.phx system is running out of memory often and swapping which causes unneeded I/O load on our storage servers. Memory should be bumped, alternatively we should see if we can somehow bring memory usage down, for example decrease number of jenkins worker threads (currently there's 5) $ free -m {{ total used free shared buff/cache available}} {{Mem: 3788 3568 102 1 117 18}} {{Swap: 3968 3968 0}} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-1959) Make usrc.py support Git tags
by Emil Natan (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1959?page=com.atlassian.jira.… ] Emil Natan reassigned OVIRT-1959: --------------------------------- Assignee: Emil Natan (was: infra) > Make usrc.py support Git tags > ----------------------------- > > Key: OVIRT-1959 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1959 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: usrc.py > Reporter: Barak Korren > Assignee: Emil Natan > > There are multiple features that could be useful to add to {{usrc.py}} that has to do with Git tags: > # *Snap to tag* - when running {{update}} take the latest tagged commit from the branch instead of the latest commit. This could further be enhanced to decide what to do when there is no tagged commit that is newer then what we already have - we can either decide not to update or update to the latest (untagged) commit. > # *Copy tag* - when running with {{--commit}} and the upstream commit we're updating into is tagged, store some information somewhere so that if the commit is merged, the CI system could tag it automatically and push that tag. It should be possible to customise the tag we add to allow it to be different but derived from the upstream tag. One possible place to store the needed information is to use a header in the commit message like we do for auto-merge. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2240) Automatically update suite's reposync-config and send a patch to OST
by Eyal Edri (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2240?page=com.atlassian.jira.… ] Eyal Edri reassigned OVIRT-2240: -------------------------------- Assignee: Galit Rosenthal (was: infra) > Automatically update suite's reposync-config and send a patch to OST > -------------------------------------------------------------------- > > Key: OVIRT-2240 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2240 > Project: oVirt - virtualization made easy > Issue Type: Epic > Components: OST > Reporter: Gal Ben Haim > Assignee: Galit Rosenthal > Labels: grooming > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2240) Automatically update suite's reposync-config and send a patch to OST
by Eyal Edri (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2240?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2240: ---------------------------------- i don't see anything in progress here, can you update the current tickets to reflect the current status? > Automatically update suite's reposync-config and send a patch to OST > -------------------------------------------------------------------- > > Key: OVIRT-2240 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2240 > Project: oVirt - virtualization made easy > Issue Type: Epic > Components: OST > Reporter: Gal Ben Haim > Assignee: infra > Labels: grooming > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2413) Set up CNV Jenkins to run STDCI
by Eyal Edri (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2413?page=com.atlassian.jira.… ] Eyal Edri deleted OVIRT-2413: ----------------------------- > Set up CNV Jenkins to run STDCI > ------------------------------- > > Key: OVIRT-2413 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2413 > Project: oVirt - virtualization made easy > Issue Type: Epic > Reporter: Daniel Belenky > Assignee: infra > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2413) Set up CNV Jenkins to run STDCI
by Eyal Edri (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2413?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-2413: ----------------------------- Description: was: Add here all tasks that needs to be done in order to run STDCI on CNV Jenkins. https://cnv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/ > Set up CNV Jenkins to run STDCI > ------------------------------- > > Key: OVIRT-2413 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2413 > Project: oVirt - virtualization made easy > Issue Type: Epic > Reporter: Daniel Belenky > Assignee: infra > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

oVirt infra weekly meeting notes 29.08.2018
by Evgheni Dereveanchin 29 Aug '18

29 Aug '18

Hi everyone, Please find this week's infra updates below: PHX: - Storage01 added to PHX engine and some VMs placed on it OVIRT-2420 <https://ovirt-jira.atlassian.net/browse/OVIRT-2420> - Docs about storage updated on ReadTheDocs <https://ovirt-infra-docs.readthedocs.io/en/latest/Phoenix_Lab/Storage_Hosts…> - PHX mirrors added to monitoring OVIRT-1303 <https://ovirt-jira.atlassian.net/browse/OVIRT-1303> - HE deploy bug still not solved but progressing BZ#1613278 <https://bugzilla.redhat.com/show_bug.cgi?id=1613278> Website: - Only Piwik is now used for usage tracking Action Items: - Deploy VM for OSAS DNS OVIRT-2455 <https://ovirt-jira.atlassian.net/browse/OVIRT-2455> -- Regards, Evgheni Dereveanchin

1 0

[JIRA] (OVIRT-2455) deploy VM to act as secondary DNS for OSAS
by Evgheni Dereveanchin (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2455?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2455: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > deploy VM to act as secondary DNS for OSAS > ------------------------------------------ > > Key: OVIRT-2455 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2455 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > Priority: Low > > We have several of our resources hosted at OSAS and for increased reliability there was a request from [~duck] to host secondary OSAS DNS in PHX. Opening ticket to gather up technical requirements and provide this service. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2455) deploy VM to act as secondary DNS for OSAS
by Evgheni Dereveanchin (oVirt JIRA) 29 Aug '18

29 Aug '18

Evgheni Dereveanchin created OVIRT-2455: ------------------------------------------- Summary: deploy VM to act as secondary DNS for OSAS Key: OVIRT-2455 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2455 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Evgheni Dereveanchin Assignee: infra Priority: Low We have several of our resources hosted at OSAS and for increased reliability there was a request from [~duck] to host secondary OSAS DNS in PHX. Opening ticket to gather up technical requirements and provide this service. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[CQ]: 74b3ccd (ovirt-ansible-image-template) failed "ovirt-4.2" system tests
by oVirt Jenkins 29 Aug '18

29 Aug '18

Change 74b3ccd (ovirt-ansible-image-template) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-image-template/commit/74b3ccd42c15d0… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3005/

1 0

[JIRA] (OVIRT-2039) Make JJB able to configure all aspects of the Jenkins master
by Ehud Yonasi (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2039?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-2039: ------------------------------------ Adding the ticket link to configuration as code plugin: https://ovirt-jira.atlassian.net/browse/OVIRT-2366 > Make JJB able to configure all aspects of the Jenkins master > ------------------------------------------------------------ > > Key: OVIRT-2039 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2039 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: JJB > Reporter: Barak Korren > Assignee: infra > Labels: upstream-contribution > > There are two aspects of jenkins that JJB knows how to configure at this time: > # Jobs > # Views > There are however, other important aspects of the Jenkins configuration one typically needs to deal with: > # Plugins > # Credentials > # Global configuration setting such as global environment > # Gerrit and GitHub trigger settings > # Custom user data > It is desirable that JJB would be able to encompass all the required configuration for a given Jenkins instance. We should consider enabling it to configure the setting mentioned above and contribute that work to the upstream JJB project or distribute it as a separate plugin. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2025) Document STDCI "V1.5" poll upstream sources jobs
by Ehud Yonasi (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2025?page=com.atlassian.jira.… ] Ehud Yonasi reassigned OVIRT-2025: ---------------------------------- Assignee: Ehud Yonasi (was: infra) > Document STDCI "V1.5" poll upstream sources jobs > ------------------------------------------------ > > Key: OVIRT-2025 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2025 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: Documentation, Standard CI (Pipelines), STDCI DSL > Reporter: Daniel Belenky > Assignee: Ehud Yonasi > > Since we're about to introduce pipeline based poll-upstream-sources jobs, we need to have documentation for the process. Docs should include: > # How to create such job for a given project: implement a JJB temlpate > # How to configure STDCI configuration file so poll-upstream-sources will work. > If anything else is needed to be documented regarding this feature, please add it as a comment. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-1989) remove old Fedora mirrors
by Evgheni Dereveanchin (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1989?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-1989: ---------------------------------------- Resolution: Done Status: Done (was: To Do) Indeed, fc25 and fc26 mirrors are removed so closing this as resolved. > remove old Fedora mirrors > ------------------------- > > Key: OVIRT-1989 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1989 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Low > > Fedora 28 is out, opening the ticket to review and remove old mirrors of previous versions. > Current Fedora mirrors and their disk usage: > |38G|/var/www/html/repos/yum/fedora-base-fc25| > |22G |/var/www/html/repos/yum/fedora-updates-fc25| > |68G |/var/www/html/repos/yum/fedora-base-fc26| > |66G |/var/www/html/repos/yum/fedora-updates-fc26| > |48G |/var/www/html/repos/yum/fedora-base-fc27| > |62G |/var/www/html/repos/yum/fedora-updates-fc27| > |92G |/var/www/html/repos/yum/fedora-base-fcraw| > We do not seem to have any Fedora 25 jobs left in CI so at least that version can be easily removed. Also worth flushing fcraw as it likely contains lots of outdated package versions. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-1996) Remove the 4.1 nightly publisher job
by Ehud Yonasi (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1996?page=com.atlassian.jira.… ] Ehud Yonasi reassigned OVIRT-1996: ---------------------------------- Assignee: Ehud Yonasi (was: infra) > Remove the 4.1 nightly publisher job > ------------------------------------ > > Key: OVIRT-1996 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1996 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Publisher jobs > Reporter: Barak Korren > Assignee: Ehud Yonasi > Labels: first_time_task > > It may be the case that we don't need the 4.1 publisher jobs any more. > Since those jobs are still in the old pre-CQ format where artifacts we collected by the job itself, their existence makes the CI system's maintenance harder... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-1996) Remove the 4.1 nightly publisher job
by Ehud Yonasi (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1996?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-1996: ------------------------------------ patch sent: https://gerrit.ovirt.org/#/c/93988/ > Remove the 4.1 nightly publisher job > ------------------------------------ > > Key: OVIRT-1996 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1996 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Publisher jobs > Reporter: Barak Korren > Assignee: infra > Labels: first_time_task > > It may be the case that we don't need the 4.1 publisher jobs any more. > Since those jobs are still in the old pre-CQ format where artifacts we collected by the job itself, their existence makes the CI system's maintenance harder... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-1989) remove old Fedora mirrors
by Ehud Yonasi (oVirt JIRA) 29 Aug '18

29 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1989?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-1989: ------------------------------------ Currently exists fc27, fc28 and fcraw. > remove old Fedora mirrors > ------------------------- > > Key: OVIRT-1989 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1989 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Low > > Fedora 28 is out, opening the ticket to review and remove old mirrors of previous versions. > Current Fedora mirrors and their disk usage: > |38G|/var/www/html/repos/yum/fedora-base-fc25| > |22G |/var/www/html/repos/yum/fedora-updates-fc25| > |68G |/var/www/html/repos/yum/fedora-base-fc26| > |66G |/var/www/html/repos/yum/fedora-updates-fc26| > |48G |/var/www/html/repos/yum/fedora-base-fc27| > |62G |/var/www/html/repos/yum/fedora-updates-fc27| > |92G |/var/www/html/repos/yum/fedora-base-fcraw| > We do not seem to have any Fedora 25 jobs left in CI so at least that version can be easily removed. Also worth flushing fcraw as it likely contains lots of outdated package versions. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1890
by jenkins＠jenkins.phx.ovirt.org 29 Aug '18

29 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d Commit message: "adding kubevirt/containerized-data-importer to stdci v2" > git rev-list --no-walk 51d7035d8a77fc8a75da5d4c064a3b1a29e82e5d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5048808394290306976.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7/gen error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7/gen' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 322 - Failure!
by jenkins＠jenkins.phx.ovirt.org 29 Aug '18

29 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/322/ Build Number: 322 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #322 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- No tests ran.

1 5

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-4.2 - Build # 260 - Failure!
by jenkins＠jenkins.phx.ovirt.org 29 Aug '18

29 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.2/26… Build Number: 260 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #260 [Milan Zamazal] ovf_import test [Eyal Edri] adding kubevirt/containerized-data-importer to stdci v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 566
by jenkins＠jenkins.phx.ovirt.org 29 Aug '18

29 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/566//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 422 - Failure!
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/422/ Build Number: 422 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #422 [Milan Zamazal] ovf_import test ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.add_disk Error Message: status: 500 reason: Internal Server Error detail: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2ecc5710c8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 122, in <lambda> api.vms.get(VM0_NAME).disks.get(DISK0_NAME).status.state == 'ok' File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35733, in get headers={"All-Content":all_content} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 46, in get return self.request(method='GET', url=url, headers=headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 500 reason: Internal Server Error detail: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 121, in add_disk lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 282, in assert_true_within_short assert_equals_within_short(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 266, in assert_equals_within_short func, value, SHORT_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 122, in <lambda> api.vms.get(VM0_NAME).disks.get(DISK0_NAME).status.state == 'ok' File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35733, in get headers={"All-Content":all_content} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 46, in get return self.request(method='GET', url=url, headers=headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) '\r\nstatus: 500\r\nreason: Internal Server Error\r\ndetail: \n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n root@localhost to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n</body></html>\n\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2ecc5710c8>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 122, in <lambda>\n api.vms.get(VM0_NAME).disks.get(DISK0_NAME).status.state == \'ok\'\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 35733, in get\n headers={"All-Content":all_content}\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 46, in get\n return self.request(method=\'GET\', url=url, headers=headers, cls=cls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request\n persistent_auth=self.__persistent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request\n persistent_auth)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request\n raise errors.RequestError(response_code, response_reason, response_body)\nRequestError: \r\nstatus: 500\r\nreason: Internal Server Error\r\ndetail: \n<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n root@localhost to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n</body></html>\n\n--------------------- >> end captured logging << ---------------------'

1 1

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1807
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1807/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" > git rev-list --no-walk 8aa2a8538d0bc192415e0eac6c831cbe94d68793 # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4357038975059197169.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2454) fcraw mirror sync job failing for several days
by Eyal Edri (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2454?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2454: ---------------------------------- Do we still need it? I don't recall any project still uses it. > fcraw mirror sync job failing for several days > ---------------------------------------------- > > Key: OVIRT-2454 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2454 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > > The FC Rawhide sync joob is failing for more than a week: > https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fcraw-x86_64/ > This brings unnecessary load to the mirror server so opening a ticket to investigate the reason -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2454) fcraw mirror sync job failing for several days
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

Evgheni Dereveanchin created OVIRT-2454: ------------------------------------------- Summary: fcraw mirror sync job failing for several days Key: OVIRT-2454 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2454 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra The FC Rawhide sync joob is failing for more than a week: https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fcraw-x86_64/ This brings unnecessary load to the mirror server so opening a ticket to investigate the reason -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2452) Re: New IP addresses for mirror ftp.plusline.net
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2452?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2452: --------------------------------------------- Request logged to white-list the new IP. Will update you once that is done. > Re: New IP addresses for mirror ftp.plusline.net > ------------------------------------------------ > > Key: OVIRT-2452 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2452 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Eyal Edri > Assignee: Evgheni Dereveanchin > > Evgheni, can you help? > Also forwarding to infra-support to open a JIRA ticket. > On Sat, Aug 25, 2018 at 7:07 PM Jan Siml <jsiml(a)plusline.net> wrote: > > Hello, > > > > we had to rebuild our mirror server and it is now available at new IP > > addresses. Since these are also used for the synchronization of > > resources.ovirt.org, I would like to ask you to unlock the new IP > > addresses. They are as follows: > > > > 82.98.66.54 > > 2a02:2e0:414:102:d::54 > > > > The SSH key used remains the same. > > > > Kind regards > > > > Jan > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/F4VSLEZLX5KMR… > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2452) Re: New IP addresses for mirror ftp.plusline.net
by Evgheni Dereveanchin (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2452?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2452: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Re: New IP addresses for mirror ftp.plusline.net > ------------------------------------------------ > > Key: OVIRT-2452 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2452 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Eyal Edri > Assignee: Evgheni Dereveanchin > > Evgheni, can you help? > Also forwarding to infra-support to open a JIRA ticket. > On Sat, Aug 25, 2018 at 7:07 PM Jan Siml <jsiml(a)plusline.net> wrote: > > Hello, > > > > we had to rebuild our mirror server and it is now available at new IP > > addresses. Since these are also used for the synchronization of > > resources.ovirt.org, I would like to ask you to unlock the new IP > > addresses. They are as follows: > > > > 82.98.66.54 > > 2a02:2e0:414:102:d::54 > > > > The SSH key used remains the same. > > > > Kind regards > > > > Jan > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/F4VSLEZLX5KMR… > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Eyal Edri (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2453?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2453: ---------------------------------- [~dbelenky(a)redhat.com] [~gbenhaim(a)redhat.com] is it possible to try running it on the new containers? > vdsm_{master,4.2} check-merge to run on bare metal slaves on CI > --------------------------------------------------------------- > > Key: OVIRT-2453 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: infra > > Hello, > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > Could you please run this job on a bare metal slave? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-fc28-x86_64 - Build # 34 - Failure!
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build Number: 34 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #34 [Yuval Turgeman] stdci: move build-artifacts to a v1 job ----------------- Failed Tests: ----------------- No tests ran.

1 2

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Daniel Belenky (oVirt JIRA) 28 Aug '18

28 Aug '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2453?page=com.atlassian.jira.… ] Daniel Belenky commented on OVIRT-2453: --------------------------------------- On Tue, Aug 28, 2018 at 8:23 AM Edward Haas <ehaas(a)redhat.com> wrote: > Hello, > > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > > Could you please run this job on a bare metal slave? > > Thanks, > Edy. > -- DANIEL BELENKY > vdsm_{master,4.2} check-merge to run on bare metal slaves on CI > --------------------------------------------------------------- > > Key: OVIRT-2453 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Edward Haas > Assignee: infra > > Hello, > VDSM check-merge is using lago VM/s to run functional tests, mainly in the > network domain. > Unfortunately, when running on a VM slave, the tests are running in a > nested VM which seems to be unstable (the nested VM is getting > stuck/crashes). > Here are some recent failures: > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ > http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ > Could you please run this job on a bare metal slave? > Thanks, > Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1886
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" > git rev-list --no-walk 8aa2a8538d0bc192415e0eac6c831cbe94d68793 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2807930280698784854.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2453) vdsm_{master,4.2} check-merge to run on bare metal slaves on CI
by Edward Haas (oVirt JIRA) 28 Aug '18

28 Aug '18

Edward Haas created OVIRT-2453: ---------------------------------- Summary: vdsm_{master,4.2} check-merge to run on bare metal slaves on CI Key: OVIRT-2453 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2453 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Edward Haas Assignee: infra Hello, VDSM check-merge is using lago VM/s to run functional tests, mainly in the network domain. Unfortunately, when running on a VM slave, the tests are running in a nested VM which seems to be unstable (the nested VM is getting stuck/crashes). Here are some recent failures: http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/247/ http://jenkins.ovirt.org/job/vdsm_4.2_check-merged-el7-x86_64/249/ Could you please run this job on a bare metal slave? Thanks, Edy. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100090)

1 0

Build failed in Jenkins: system-sync_mirrors-glusterfs-4.1-test-el7-x86_64 #1
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> ------------------------------------------ Started by user Ehud Yonasi [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> Cloning the remote Git repository Cloning repository http://gerrit.ovirt.org/jenkins.git > git init <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-4.1-test-el7-x86…> # timeout=10 Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 No valid HEAD. Skipping the resetting > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 8aa2a8538d0bc192415e0eac6c831cbe94d68793 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8aa2a8538d0bc192415e0eac6c831cbe94d68793 Commit message: "mock_runner: Set no_proxy for proxied configs" First time build. Skipping changelog. [system-sync_mirrors-glusterfs-4.1-test-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5612468679084601662.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-4.1-test-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Warning: cannot find repository glusterfs-4.1-test-el7 No repositories found Build step 'Execute shell' marked build as failure

2 4

oVirt infra daily report - unstable production jobs - 565
by jenkins＠jenkins.phx.ovirt.org 28 Aug '18

28 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/565//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 93925,2 (vdsm) failed "ovirt-4.2" system tests
by oVirt Jenkins 27 Aug '18

27 Aug '18

Change 93925,2 (vdsm) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93925/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2993/

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 464 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 464 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #464 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_restart_he_vm.restart_he_vm Error Message: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1: lago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0 lago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true} root: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0 lago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown lago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Command succeeded root: INFO: * Waiting for VM to be down... lago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false} lago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true} lago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true} root: INFO: * VM is down. root: INFO: * Stopping services... lago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Starting services... lago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent lago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 root: INFO: * Waiting for agent to be ready... lago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1 lago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. lago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output: !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : lago-he-basic-iscsi-suite-master-host-0.lago.local Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} Score : 0 stopped : False Local maintenance : False crc32 : 56f93df8 local_conf_timestamp : 2896 Host timestamp : 2895 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2895 (Sun Aug 26 23:29:50 2018) host-id=1 score=0 vm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018) conf_on_shared_storage=True maintenance=False state=ReinitializeFSM stopped=False --== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : lago-he-basic-iscsi-suite-master-host-1 Host ID : 2 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : ffa35d6d local_conf_timestamp : 2892 Host timestamp : 2891 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2891 (Sun Aug 26 23:29:45 2018) host-id=2 score=3400 vm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! root: INFO: * Agent is ready. root: INFO: * Starting VM... lago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output: VM exists and is down, cleaning up and restarting root: INFO: * Command succeeded root: INFO: * Waiting for VM to be UP... lago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\nhost-id=1\nscore=0\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=ReinitializeFSM\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true} lago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} root: INFO: * VM is UP. root: INFO: * Waiting for engine to start... lago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true} lago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true} lago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true} lago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true} lago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true} lago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true} lago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true} lago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true} lago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true} lago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0: lago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0 lago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) RuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 53, in restart_he_vm _wait_for_engine_health(host) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 185, in _wait_for_engine_health testlib.assert_true_within_long(lambda: any( File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda> for k, v in _get_he_status(host).items() File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status raise RuntimeError('could not parse JSON: %s' % ret.out) 'could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: end task:a3c71d1c-cb12-4fef-9e6a-d5e0c1b07550:Get ssh client for lago-he-basic-iscsi-suite-master-host-1:\nlago.ssh: DEBUG: Running 43cfc44a on lago-he-basic-iscsi-suite-master-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 returned with 0\nlago.ssh: DEBUG: Command 43cfc44a on lago-he-basic-iscsi-suite-master-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2810 (Sun Aug 26 23:28:24 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2812 (Sun Aug 26 23:28:26 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7f7e634d", "local_conf_timestamp": 2812, "host-ts": 2810}, "global_maintenance": true}\n\nroot: INFO: * Shutting down HE VM on host: lago-he-basic-iscsi-suite-master-host-0\nlago.ssh: DEBUG: start task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:42afe970-0be5-40d6-834b-245d877c0d1a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 445276b0 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-shutdown\nlago.ssh: DEBUG: Command 445276b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be down...\nlago.ssh: DEBUG: start task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:d92ec1bd-a571-4e86-b3a6-7bedd805567d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 44f97960 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 44f97960 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2805 (Sun Aug 26 23:28:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2806 (Sun Aug 26 23:28:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fd22cf78", "local_conf_timestamp": 2806, "host-ts": 2805}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2800 (Sun Aug 26 23:28:14 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2801 (Sun Aug 26 23:28:15 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "063aa920", "local_conf_timestamp": 2801, "host-ts": 2800}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a1e22b81-c1cf-4b25-a1f7-342286fbe81b:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4b960ed2 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f5948091-c497-4f3b-9761-04d42a2cc421:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 4df162c6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 4df162c6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8b462547-686e-4dbe-8bfb-91d4ba2931dd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 504ec626 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 504ec626 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2825 (Sun Aug 26 23:28:40 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2826 (Sun Aug 26 23:28:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "ca41a4cb", "local_conf_timestamp": 2826, "host-ts": 2825}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2821 (Sun Aug 26 23:28:35 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2822 (Sun Aug 26 23:28:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6bac30e9", "local_conf_timestamp": 2822, "host-ts": 2821}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f2eef89a-f42e-4c79-9337-72571575f679:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 52a4916c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 52a4916c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2835 (Sun Aug 26 23:28:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2837 (Sun Aug 26 23:28:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a4710c6", "local_conf_timestamp": 2837, "host-ts": 2835}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2831 (Sun Aug 26 23:28:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2833 (Sun Aug 26 23:28:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fce69ba8", "local_conf_timestamp": 2833, "host-ts": 2831}, "global_maintenance": true}\n\nroot: INFO: * VM is down.\nroot: INFO: * Stopping services...\nlago.ssh: DEBUG: start task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2c25f4a6-f2ef-4cfa-822f-b929c4dfd509:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0: systemctl stop vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5322b5b0 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Starting services...\nlago.ssh: DEBUG: start task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ee6b4609-b5ac-4f3c-a182-7ed44b8deeb6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0: systemctl start vdsmd ovirt-ha-broker ovirt-ha-agent\nlago.ssh: DEBUG: Command 5a5b1dd6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nroot: INFO: * Waiting for agent to be ready...\nlago.ssh: DEBUG: start task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:facd8e55-61ed-4fb7-b9fe-c3ad705d968a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 5c4f3d84 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3eb14b7c-074b-4c11-a117-392f8b162c2e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 62a124ea on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 62a124ea on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c78b3a53-f7bb-480e-839e-b57e9ca7eb4d:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 64b4fa68 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca525d74-b2b6-4297-ad54-b044bc452568:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 66d48ffc on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 66d48ffc on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:f871d6fb-e79f-477b-b5ca-99b939a4d9f7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 69018c26 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 69018c26 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:b86743b4-04cf-4455-aba9-0bc61ad3815e:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6b1f2dce on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8e87c593-e4ec-4db5-b533-eab312a2c1cb:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6d468714 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6d468714 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:221b1e18-be90-427f-8a54-9291fe6f5959:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 6f693f32 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 6f693f32 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:52fbd8f8-8d38-4465-9ecc-c00e381b7563:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 71a1db42 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 71a1db42 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7a8d4a8c-12b5-4a68-b436-535b8d09f517:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 returned with 1\nlago.ssh: DEBUG: Command 73c0ed14 on lago-he-basic-iscsi-suite-master-host-0 output:\n The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.\n\nlago.ssh: DEBUG: start task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3de81b6a-badb-4222-8aca-128ebdfec400:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 75dd4066 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 75dd4066 on lago-he-basic-iscsi-suite-master-host-0 output:\n \n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-0.lago.local (id: 1) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : True\nHostname : lago-he-basic-iscsi-suite-master-host-0.lago.local\nHost ID : 1\nEngine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}\nScore : 0\nstopped : False\nLocal maintenance : False\ncrc32 : 56f93df8\nlocal_conf_timestamp : 2896\nHost timestamp : 2895\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2895 (Sun Aug 26 23:29:50 2018)\n\thost-id=1\n\tscore=0\n\tvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=ReinitializeFSM\n\tstopped=False\n\n\n--== Host lago-he-basic-iscsi-suite-master-host-1 (id: 2) status ==--\n\nconf_on_shared_storage : True\nStatus up-to-date : False\nHostname : lago-he-basic-iscsi-suite-master-host-1\nHost ID : 2\nEngine status : unknown stale-data\nScore : 3400\nstopped : False\nLocal maintenance : False\ncrc32 : ffa35d6d\nlocal_conf_timestamp : 2892\nHost timestamp : 2891\nExtra metadata (valid at timestamp):\n\tmetadata_parse_version=1\n\tmetadata_feature_version=1\n\ttimestamp=2891 (Sun Aug 26 23:29:45 2018)\n\thost-id=2\n\tscore=3400\n\tvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\n\tconf_on_shared_storage=True\n\tmaintenance=False\n\tstate=GlobalMaintenance\n\tstopped=False\n\n\n!! Cluster is in GLOBAL MAINTENANCE mode !!\n\n\nroot: INFO: * Agent is ready.\nroot: INFO: * Starting VM...\nlago.ssh: DEBUG: start task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:0174ef46-3f09-4c57-b0cc-135fc059665a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 764e856e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-start\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 764e856e on lago-he-basic-iscsi-suite-master-host-0 output:\n VM exists and is down, cleaning up and restarting\n\nroot: INFO: * Command succeeded\nroot: INFO: * Waiting for VM to be UP...\nlago.ssh: DEBUG: start task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:30ee44ce-7f0b-4b16-ba1b-7b6964891efd:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 77b7ccda on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 77b7ccda on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2895 (Sun Aug 26 23:29:50 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2896 (Sun Aug 26 23:29:51 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=ReinitializeFSM\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"}, "score": 0, "stopped": false, "maintenance": false, "crc32": "56f93df8", "local_conf_timestamp": 2896, "host-ts": 2895}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2891 (Sun Aug 26 23:29:45 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2892 (Sun Aug 26 23:29:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ffa35d6d", "local_conf_timestamp": 2892, "host-ts": 2891}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ef26fadc-fd32-4afb-9a06-07e13a3e556f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7e2baca8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nroot: INFO: * VM is UP.\nroot: INFO: * Waiting for engine to start...\nlago.ssh: DEBUG: start task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:22f9ce85-5236-43de-956e-60eedabdee44:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 7ea960f8 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2905 (Sun Aug 26 23:30:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2906 (Sun Aug 26 23:30:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "9fce1b92", "local_conf_timestamp": 2906, "host-ts": 2905}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2901 (Sun Aug 26 23:29:55 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2902 (Sun Aug 26 23:29:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "68110bea", "local_conf_timestamp": 2902, "host-ts": 2901}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:10d11112-af95-4062-82b5-90a12379d806:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 85234b10 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 85234b10 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2911 (Sun Aug 26 23:30:05 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2913 (Sun Aug 26 23:30:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ff5ba0ab", "local_conf_timestamp": 2913, "host-ts": 2911}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8600ff29-4795-4922-9bb5-1ea8cb47aff6:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8778cab6 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8778cab6 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2915 (Sun Aug 26 23:30:10 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2916 (Sun Aug 26 23:30:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "69a0d34f", "local_conf_timestamp": 2916, "host-ts": 2915}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:6251e2ec-36ea-42cd-a7cf-018d3efaf73a:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 89e10548 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 89e10548 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:5671a60c-09fc-4f60-97a2-270f9f5b4ca1:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8c4fef60 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2925 (Sun Aug 26 23:30:20 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2927 (Sun Aug 26 23:30:22 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "df65bcff", "local_conf_timestamp": 2927, "host-ts": 2925}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2922 (Sun Aug 26 23:30:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2923 (Sun Aug 26 23:30:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ee130676", "local_conf_timestamp": 2923, "host-ts": 2922}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ca12f761-8a01-4613-8217-3848165adb82:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 8efac10e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 8efac10e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:2ce2a039-af5f-47e2-b0c8-650b8ca40276:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 915ac1c4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2935 (Sun Aug 26 23:30:30 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2937 (Sun Aug 26 23:30:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "41a00e65", "local_conf_timestamp": 2937, "host-ts": 2935}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2932 (Sun Aug 26 23:30:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2934 (Sun Aug 26 23:30:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "903a0802", "local_conf_timestamp": 2934, "host-ts": 2932}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:50565f8c-6b24-4ae7-9eba-bbf9b31b3867:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 93ca2012 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 93ca2012 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8de05a79-ab0e-4659-bc5c-03ab090af777:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 964c95f4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 964c95f4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c85287ef-a0b5-4cea-ab1a-cfc1c4cc97d4:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 98aaf00c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 98aaf00c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2945 (Sun Aug 26 23:30:40 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2946 (Sun Aug 26 23:30:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "fad4f715", "local_conf_timestamp": 2946, "host-ts": 2945}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2942 (Sun Aug 26 23:30:36 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2943 (Sun Aug 26 23:30:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c22d5447", "local_conf_timestamp": 2943, "host-ts": 2942}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:c1aa3391-00ad-4c4a-a698-304aa8d8e598:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9b10536e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9b10536e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:a7f2e7dd-ffdc-4265-a6b0-58c16e4d076f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9d796f1e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9d796f1e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2955 (Sun Aug 26 23:30:50 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2957 (Sun Aug 26 23:30:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "6d9e5c54", "local_conf_timestamp": 2957, "host-ts": 2955}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2952 (Sun Aug 26 23:30:46 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2953 (Sun Aug 26 23:30:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "2260cf90", "local_conf_timestamp": 2953, "host-ts": 2952}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:197b6456-4573-4fe5-a902-4d4135afd1c5:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running 9fef209a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command 9fef209a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:e449730a-24b3-4ba7-a32a-9841d681e8d7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a251ea20 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a251ea20 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:8fa960e8-d3d3-4de3-8673-7ccb2e8e3232:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a4b99e3e on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a4b99e3e on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2965 (Sun Aug 26 23:31:00 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=2966 (Sun Aug 26 23:31:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e13ec6fa", "local_conf_timestamp": 2966, "host-ts": 2965}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2962 (Sun Aug 26 23:30:56 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2963 (Sun Aug 26 23:30:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "d9c765a8", "local_conf_timestamp": 2963, "host-ts": 2962}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:df444f3d-0958-4232-bbe2-79c9b53d6df7:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a7386262 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a7386262 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ccb637f6-e93c-474f-8822-99fa3e47e492:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command a9b0a9b4 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2975 (Sun Aug 26 23:31:10 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2976 (Sun Aug 26 23:31:11 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "1cb13a70", "local_conf_timestamp": 2976, "host-ts": 2975}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2972 (Sun Aug 26 23:31:06 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2973 (Sun Aug 26 23:31:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "398afe7f", "local_conf_timestamp": 2973, "host-ts": 2972}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c133bbf-6ccf-4f97-a708-4902c159a68f:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ac14d18a on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ac14d18a on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:ff5cd16b-0c57-4df9-bbcc-9fd94b5fccb8:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command ae7c4e6c on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:7fcd4b38-7d15-4000-ad18-d363a3d3a7ab:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b0c63368 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b0c63368 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2985 (Sun Aug 26 23:31:20 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2986 (Sun Aug 26 23:31:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "bf6a342a", "local_conf_timestamp": 2986, "host-ts": 2985}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2982 (Sun Aug 26 23:31:16 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2984 (Sun Aug 26 23:31:18 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "04356586", "local_conf_timestamp": 2984, "host-ts": 2982}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:3c20a1c0-ec61-41b6-b247-25ac0093a494:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b30f7738 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b30f7738 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "7a1c6bf2", "local_conf_timestamp": 2993, "host-ts": 2992}, "global_maintenance": true}\n\nlago.ssh: DEBUG: start task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: end task:90657a8e-cbc1-4061-823a-c2e3d71635c3:Get ssh client for lago-he-basic-iscsi-suite-master-host-0:\nlago.ssh: DEBUG: Running b555e324 on lago-he-basic-iscsi-suite-master-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 returned with 0\nlago.ssh: DEBUG: Command b555e324 on lago-he-basic-iscsi-suite-master-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5d8fad1410>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 187, in <lambda>\n for k, v in _get_he_status(host).items()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master/ovirt-system-tests/he-basic-iscsi-suite-master/test-scenarios/008_restart_he_vm.py", line 128, in _get_he_status\n raise RuntimeError(\'could not parse JSON: %s\' % ret.out)\nRuntimeError: could not parse JSON: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2995 (Sun Aug 26 23:31:30 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=2997 (Sun Aug 26 23:31:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "28209f6b", "local_conf_timestamp": 2997, "host-ts": 2995}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2992 (Sun Aug 26 23:31:26 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=2993 (Sun Aug 26 23:31:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=GlobalMaintenance\\nstopped=False\\n", "hostname": "lago-he-basic-iscsi-suite-master-host-1",\n--------------------- >> end captured logging << ---------------------'

1 2

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1884
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1863032023757664663.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 639 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/639/ Build Number: 639 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #639 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.

1 1

[CQ]: 93756,2 (imgbased) failed "ovirt-4.2" system tests
by oVirt Jenkins 27 Aug '18

27 Aug '18

Change 93756,2 (imgbased) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/93756/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2986/

1 0

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 255 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/255/ Build Number: 255 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #255 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 012_local_maintenance_sdk.local_maintenance Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 45, in local_maintenance vm_service = test_utils.get_vm_service(engine, VM_HE_NAME) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 71, in __call__ value = self.func(*args) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/test_utils/__init__.py", line 101, in get_vm_service vm = vms_service.list(search='name={}'.format(vm_name))[0] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 34154, in list return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) "The response content type 'text/html; charset=iso-8859-1' isn't the expected XML\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\n--------------------- >> end captured logging << ---------------------"

1 1

[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64 - Build # 40 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build Number: 40 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #40 [Yuval Turgeman] automation: remove build-artifacts* from stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #1880
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5224362443311299032.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 3

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.2 - Build # 510 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.2/510/ Build Number: 510 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #510 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 006_network_by_label.assign_hosts_network_label Error Message: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 65, in assign_hosts_network_label nt.assert_true(all(vt.join_all())) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label host_nic=nic File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) '\r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.2/ovirt-system-tests/he-basic-suite-4.2/test-scenarios/006_network_by_label.py", line 57, in _assign_host_network_label\n host_nic=nic\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16231, in add\n headers={"Correlation-Id":correlation_id, "Expect":expect}\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add\n return self.request(\'POST\', url, body, headers, cls=cls)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request\n persistent_auth=self.__persistent_auth\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request\n persistent_auth)\n File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request\n raise errors.RequestError(response_code, response_reason, response_body)\nRequestError: \r\nstatus: 409\r\nreason: Conflict\r\ndetail: Cannot add Label. Another Setup Networks or Host Refresh process in progress on the host. Please try later.\n--------------------- >> end captured logging << ---------------------'

1 2

Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #1802
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/1802/displ…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3afc9ba57755a1b2574f7a177184eb66d5f6a90d (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3afc9ba57755a1b2574f7a177184eb66d5f6a90d Commit message: "ovirt-release: drop non working jobs" > git rev-list --no-walk 3afc9ba57755a1b2574f7a177184eb66d5f6a90d # timeout=10 [system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4293223092552671319.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Resyncing repo: epel-el6 Syncing yum repo epel-el6 for arch: x86_64 epel-el6/group | 303 kB 00:00 tklib-0.5-5.el6.noarch.rpm 0% [ ] 0.0 B/s | 0 B --:-- ETA tklib-0.5-5.el6.noarch.rpm | 416 kB 00:02 Generating yum repo metadata for: epel-el6 Worker exited with non-zero value: -9. Fatal. Worker exited with non-zero value: -9. Fatal. Spawning worker 0 with 1519 pkgs Spawning worker 1 with 1519 pkgs Spawning worker 2 with 1519 pkgs Spawning worker 3 with 1518 pkgs Spawning worker 4 with 1518 pkgs Spawning worker 5 with 1518 pkgs Spawning worker 6 with 1518 pkgs Spawning worker 7 with 1518 pkgs Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 564
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/564//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 637 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Aug '18

27 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/637/ Build Number: 637 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #637 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f031a4cfed8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f031a4cfed8>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 36676, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-master - Build # 318 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18

26 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-master/318/ Build Number: 318 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #318 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- No tests ran.

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 462 - Failure!
by jenkins＠jenkins.phx.ovirt.org 26 Aug '18

26 Aug '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 462 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #462 [Roy Golan] Introduce openshift-on-ovirt ----------------- Failed Tests: ----------------- No tests ran.

1 1