Re: [ovirt-users] oVirt fails to install on nodes after moving engine to C7 host

6 Sep 2016

      Thanks for the info. It looks like everything is tied together (as it
should be) dependency wise, so downgrading otopi isn't going to be that
simple.

[root@engine ~]# yum downgrade otopi-1.4.2-1.el7.centos
otopi-java-1.4.2-1.el7.centos
<snip>
Resolving Dependencies
--> Running transaction check
---> Package otopi.noarch 0:1.4.2-1.el7.centos will be a downgrade
---> Package otopi.noarch 0:1.5.2-1.el7.centos will be erased
---> Package otopi-java.noarch 0:1.4.2-1.el7.centos will be a downgrade
---> Package otopi-java.noarch 0:1.5.2-1.el7.centos will be erased
--> Finished Dependency Resolution
Error: Package: ovirt-engine-setup-base-4.0.2.7-1.el7.centos.noarch
(@ovirt-4.0)
           Requires: otopi >= 1.5.0
           Removing: otopi-1.5.2-1.el7.centos.noarch (@ovirt-4.0)
               otopi = 1.5.2-1.el7.centos
           Downgraded By: otopi-1.4.2-1.el7.centos.noarch (ovirt-3.6)
               otopi = 1.4.2-1.el7.centos
           Available: otopi-1.4.0-1.el7.noarch (centos-ovirt36)
               otopi = 1.4.0-1.el7
           Available: otopi-1.4.0-1.el7.centos.noarch (ovirt-3.6)
               otopi = 1.4.0-1.el7.centos
           Available: otopi-1.4.1-1.el7.noarch (centos-ovirt36)
               otopi = 1.4.1-1.el7
           Available: otopi-1.4.1-1.el7.centos.noarch (ovirt-3.6)
               otopi = 1.4.1-1.el7.centos
           Available: otopi-1.5.0-1.el7.centos.noarch (ovirt-4.0)
               otopi = 1.5.0-1.el7.centos
           Available: otopi-1.5.1-1.el7.noarch (centos-ovirt40-release)
               otopi = 1.5.1-1.el7
           Available: otopi-1.5.1-1.el7.centos.noarch (ovirt-4.0)
               otopi = 1.5.1-1.el7.centos
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@engine ~]#

Which just goes down a rabbit hole of dependency chasing for most ovirt*
packages. I can't think of a good way past this apart from a reinstall?

On Tue, 6 Sep 2016, at 06:19 PM, Yedidyah Bar David wrote:
...
On Tue, Sep 6, 2016 at 10:43 AM, James <mailinglists@mooash.com> wrote:
...
Thanks for your help! I've pasted the log lines requested below. Also
worth noting that I tried upgrading it to 4.0 which I think has lead to
some broken package versions. Everything seems to work, but there are
some 4.0 packages installed.
Perhaps that's the problem [1]. Can you try downgrading otopi to 1.4 and
remove /var/cache/ovirt-engine/ovirt-host-deploy.tar on engine machine?
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1348091
Your package state is not very healthy but if above is enough to get you
to install a host, and later you succeed upgrading to 4.0, you should be
ok.
...
When trying to install 4.0 I followed the docs (yum update
engine-setup*) however when running engine-setup it informed me that it
couldn't take the install to 4.0 because of the 3.5 cluster that
existed. Currently installed pacakges:
Installed Packages
ovirt-engine.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-backend.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-cli.noarch
                           3.6.8.0-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-dbscripts.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-dwh.noarch
                           3.6.6-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-dwh-setup.noarch
                           4.0.1-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-extension-aaa-jdbc.noarch
                           1.0.7-1.el7
                                         @ovirt-3.6
ovirt-engine-extension-aaa-ldap.noarch
                           1.2.1-1.el7
                                         @ovirt-4.0
ovirt-engine-extension-aaa-ldap-setup.noarch
                           1.2.1-1.el7
                                         @ovirt-4.0
ovirt-engine-extensions-api-impl.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-lib.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-restapi.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-sdk-python.noarch
                           3.6.8.0-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup-base.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup-plugin-ovirt-engine.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup-plugin-ovirt-engine-common.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-setup-plugin-websocket-proxy.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-tools.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-tools-backup.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-userportal.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-vmconsole-proxy-helper.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-webadmin-portal.noarch
                           3.6.7.5-1.el7.centos
                                         @ovirt-3.6
ovirt-engine-websocket-proxy.noarch
                           4.0.2.7-1.el7.centos
                                         @ovirt-4.0
ovirt-engine-wildfly.x86_64
                           8.2.1-1.el7
                                         @ovirt-3.6
ovirt-engine-wildfly-overlay.noarch
                           8.0.5-1.el7
                                         @ovirt-3.6
ovirt-host-deploy.noarch
                           1.4.1-1.el7.centos
                                         @ovirt-3.6
ovirt-host-deploy-java.noarch
                           1.4.1-1.el7.centos
                                         @ovirt-3.6
ovirt-image-uploader.noarch
                           3.6.0-1.el7.centos
                                         @ovirt-3.6
ovirt-imageio-proxy-setup.noarch
                           0.3.0-0.201606191345.git9f3d6d4.el7.centos
                                         @ovirt-4.0
ovirt-iso-uploader.noarch
                           3.6.0-1.el7.centos
                                         @ovirt-3.6
ovirt-release35.noarch
                           006-1
                                         installed
ovirt-release36.noarch
                           1:3.6.7-1
                                         @/ovirt-release36
ovirt-release40.noarch
                           4.0.3-1
                                         @/ovirt-release40
ovirt-setup-lib.noarch
                           1.0.1-1.el7.centos
                                         @ovirt-3.6
ovirt-vmconsole.noarch
                           1.0.2-1.el7.centos
                                         @ovirt-3.6
ovirt-vmconsole-proxy.noarch
                           1.0.2-1.el7.centos
                                         @ovirt-3.6
Is it safe to assume that the packages just aren't happy and a
backup/restore is the best option at this point? Or is this still
salvageable?
2016-09-06 15:12:14,682 INFO
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Running command:
InstallVdsInternalCommand internal: true. Entities affected :  ID:
12489166-d1ea-4bcc-8e96-526a1826c506 Type: VDS
2016-09-06 15:12:14,682 INFO
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Before Installation host
12489166-d1ea-4bcc-8e96-526a1826c506, srv3.xxx.xxx.xxx
2016-09-06 15:12:14,715 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] START,
SetVdsStatusVDSCommand(HostName = srv3.xxx.xxx.xxx,
SetVdsStatusVDSCommandParameters:{runAsync='true',
hostId='12489166-d1ea-4bcc-8e96-526a1826c506', status='Installing',
nonOperationalReason='NONE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: 23376707
2016-09-06 15:12:14,719 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] FINISH,
SetVdsStatusVDSCommand, log id: 23376707
2016-09-06 15:12:14,803 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56,
Call Stack: null, Custom Event ID: -1, Message: Installing Host
srv3.xxx.xxx.xxx. Connected to host <host ip> with SSH key fingerprint:
SHA256:<redacted>.
2016-09-06 15:12:14,824 INFO
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Installation of <host ip>.
Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}"
mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" >
/dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
--warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy
DIALOG/dialect=str:machine DIALOG/customization=bool:True <
/var/cache/ovirt-engine/ovirt-host-deploy.tar
2016-09-06 15:12:14,824 INFO
[org.ovirt.engine.core.utils.archivers.tar.CachedTar]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Tarball
'/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh
2016-09-06 15:12:14,852 INFO
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] SSH execute 'root@<host
ip>' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1;
rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
"${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy
DIALOG/dialect=str:machine DIALOG/customization=bool:True'
2016-09-06 15:12:15,125 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] SSH error running command
root@<host ip>:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d
-t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
--warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy
DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command
returned failure code 1 during SSH session 'root@<host ip>'
2016-09-06 15:12:15,121 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy)
[2bfaf56] Error during deploy dialog: java.io.IOException: Unexpected
connection termination
        at
        org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:387)
        [otopi.jar:]
        at
        org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:404)
        [otopi.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase._threadMain(VdsDeployBase.java:304)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.access$800(VdsDeployBase.java:45)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase$12.run(VdsDeployBase.java:386)
        [bll.jar:]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
2016-09-06 15:12:15,126 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Exception:
java.io.IOException: Command returned failure code 1 during SSH session
'root@<host ip>'
        at
        org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527)
        [uutils.jar:]
        at
        org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312)
        [uutils.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
        [utils.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
        [utils.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89)
        [utils.jar:]
        at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
2016-09-06 15:12:15,127 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Error during host <host
ip> install: java.io.IOException: Command returned failure code 1 during
SSH session 'root@<host ip>'
        at
        org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527)
        [uutils.jar:]
        at
        org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312)
        [uutils.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
        [utils.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
        [utils.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89)
        [utils.jar:]
        at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
2016-09-06 15:12:15,156 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56,
Call Stack: null, Custom Event ID: -1, Message: Failed to install Host
srv3.xxx.xxx.xxx. Command returned failure code 1 during SSH session
'root@<host ip>'.
2016-09-06 15:12:15,156 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Error during host <host
ip> install, prefering first exception: Unexpected connection
termination
2016-09-06 15:12:15,156 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Host installation failed
for host '12489166-d1ea-4bcc-8e96-526a1826c506', 'srv3.xxx.xxx.xxx':
Unexpected connection termination
2016-09-06 15:12:15,180 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] START,
SetVdsStatusVDSCommand(HostName = srv3.xxx.xxx.xxx,
SetVdsStatusVDSCommandParameters:{runAsync='true',
hostId='12489166-d1ea-4bcc-8e96-526a1826c506', status='InstallFailed',
nonOperationalReason='NONE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: 523cdcff
2016-09-06 15:12:15,184 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] FINISH,
SetVdsStatusVDSCommand, log id: 523cdcff
2016-09-06 15:12:15,190 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56,
Call Stack: null, Custom Event ID: -1, Message: Host srv3.xxx.xxx.xxx
installation failed. Unexpected connection termination.
2016-09-06 15:12:15,190 INFO
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(org.ovirt.thread.pool-8-thread-32) [2bfaf56] Lock freed to object
'EngineLock:{exclusiveLocks='[12489166-d1ea-4bcc-8e96-526a1826c506=<VDS,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-09-06 15:12:15,695 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-3) [2bfaf56] Correlation ID: null, Call Stack: null,
Custom Event ID: -1, Message: Executing power management status on Host
srv3.xxx.xxx.xxx using Proxy Host srv2.xxx.xxx.xxx and Fence Agent
ipmilan:xxx.xx.xx.xx.
2016-09-06 15:12:15,696 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-3) [2bfaf56] START, FenceVdsVDSCommand(HostName = srv2.xxx.xxx.xxx,
FenceVdsVDSCommandParameters:{runAsync='true',
hostId='d1fbefab-c5f7-4c4a-811f-0f4e2cd7d9e3',
targetVdsId='12489166-d1ea-4bcc-8e96-526a1826c506', action='STATUS',
agent='FenceAgent:{id='7cddafa7-94bd-48fe-b593-fbc3fe24950e',
hostId='12489166-d1ea-4bcc-8e96-526a1826c506', order='1',
type='ipmilan', ip='xxx.xx.xx.xx', port='null', user='root',
password='***', encryptOptions='false', options='privlvl=OPERATOR
delay=10
lanplus=1'}', policy='null'}), log id: 32d678be
2016-09-06 15:12:15,981 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-3) [2bfaf56] FINISH, FenceVdsVDSCommand, return:
FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''},
log id: 32d678be
2016-09-06 15:12:15,988 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-3) [2bfaf56] Correlation ID: 2bfaf56, Call Stack: null,
Custom Event ID: -1, Message: Host srv3.xxx.xxx.xxx configuration was
updated by xxx@xxx.xxx.
On Tue, 6 Sep 2016, at 05:01 PM, Yedidyah Bar David wrote:
...
On Tue, Sep 6, 2016 at 6:08 AM, James <mailinglists@mooash.com> wrote:
...
Hey,
I moved our standalone engine from a C6 host to a C7 host and now it
wont install/setup oVirt nodes. It just responds with messages like
this:
Host xxx.xxx.xxx installation failed. Unexpected connection termination.
Failed to install Host xxx.xxx.xxx. Command returned failure code 1
during SSH session 'root@xxx.xx.xx.x'.
And from engine.log
2016-09-06 12:20:08,011 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-41) [55397a53] SSH error running command
root@xxx.xx.xx.x:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp
-d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MY
2016-09-06 12:20:08,011 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-41) [55397a53] SSH error running command
root@xxx.xx.xx.x:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp
-d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MY
TMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
--warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy
DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command
returned failure code 1 during SSH session 'root@xxx.xx.xx.x'
2016-09-06 12:20:08,007 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy)
[55397a53] Error during deploy dialog: java.io.IOException: Unexpected
connection termination
        at
        org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:387)
        [otopi.jar:]
        at
        org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:404)
        [otopi.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase._threadMain(VdsDeployBase.java:304)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.access$800(VdsDeployBase.java:45)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase$12.run(VdsDeployBase.java:386)
        [bll.jar:]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
2016-09-06 12:20:08,013 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(org.ovirt.thread.pool-8-thread-41) [55397a53] Exception:
java.io.IOException: Command returned failure code 1 during SSH session
'root@xxx.xx.xx.x'
        at
        org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527)
        [uutils.jar:]
        at
        org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312)
        [uutils.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
        [utils.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
        [utils.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89)
        [utils.jar:]
        at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
2016-09-06 12:20:08,014 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase]
(org.ovirt.thread.pool-8-thread-41) [55397a53] Error during host
xxx.xx.xx.x install: java.io.IOException: Command returned failure code
1 during SSH session 'root@xxx.xx.xx.x'
        at
        org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527)
        [uutils.jar:]
        at
        org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312)
        [uutils.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
        [utils.jar:]
        at
        org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
        [utils.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172)
        [bll.jar:]
        at
        org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181)
        [bll.jar:]
        at
        org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89)
        [utils.jar:]
        at
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        [rt.jar:1.8.0_101]
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
I've confirmed connectivity from the engine to the hosts themselves and
it can SSH fine, even running the above command manually works fine.
To confirm, I migrated the oVirt engine by taking a backup, shutting
down the old host, changing the hostname/IP of the new host to be the
same as the old and restoring the backup. This is happening on oVirt 3.6
and is currently stopping us from being able to upgrade to oVirt 4.0
since we cannot convert one of our clusters to 3.6 compatibility mode
(its currently on 3.5) without migrating the VM hosts. It happens on a
fresh install or a reinstall, so not sure whats happening.
Can you please check/attach more of engine.log from before the error?
Start
with the line having 'Before Installation'. Thanks.
--
Didi
-- 
Didi