Re: [ovirt-users] oVirt fails to install on nodes after moving engine to C7 host

Thanks for your help! I've pasted the log lines requested below. Also worth noting that I tried upgrading it to 4.0 which I think has lead to some broken package versions. Everything seems to work, but there are some 4.0 packages installed.

When trying to install 4.0 I followed the docs (yum update engine-setup*) however when running engine-setup it informed me that it couldn't take the install to 4.0 because of the 3.5 cluster that existed. Currently installed pacakges: Installed Packages ovirt-engine.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-backend.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-cli.noarch 3.6.8.0-1.el7.centos @ovirt-4.0 ovirt-engine-dbscripts.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-dwh.noarch 3.6.6-1.el7.centos @ovirt-3.6 ovirt-engine-dwh-setup.noarch 4.0.1-1.el7.centos @ovirt-4.0 ovirt-engine-extension-aaa-jdbc.noarch 1.0.7-1.el7 @ovirt-3.6 ovirt-engine-extension-aaa-ldap.noarch 1.2.1-1.el7 @ovirt-4.0 ovirt-engine-extension-aaa-ldap-setup.noarch 1.2.1-1.el7 @ovirt-4.0 ovirt-engine-extensions-api-impl.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-lib.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-restapi.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-sdk-python.noarch 3.6.8.0-1.el7.centos @ovirt-4.0 ovirt-engine-setup.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-setup-base.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-setup-plugin-ovirt-engine.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-setup-plugin-ovirt-engine-common.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-setup-plugin-websocket-proxy.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-tools.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-tools-backup.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-userportal.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-vmconsole-proxy-helper.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-webadmin-portal.noarch 3.6.7.5-1.el7.centos @ovirt-3.6 ovirt-engine-websocket-proxy.noarch 4.0.2.7-1.el7.centos @ovirt-4.0 ovirt-engine-wildfly.x86_64 8.2.1-1.el7 @ovirt-3.6 ovirt-engine-wildfly-overlay.noarch 8.0.5-1.el7 @ovirt-3.6 ovirt-host-deploy.noarch 1.4.1-1.el7.centos @ovirt-3.6 ovirt-host-deploy-java.noarch 1.4.1-1.el7.centos @ovirt-3.6 ovirt-image-uploader.noarch 3.6.0-1.el7.centos @ovirt-3.6 ovirt-imageio-proxy-setup.noarch 0.3.0-0.201606191345.git9f3d6d4.el7.centos @ovirt-4.0 ovirt-iso-uploader.noarch 3.6.0-1.el7.centos @ovirt-3.6 ovirt-release35.noarch 006-1 installed ovirt-release36.noarch 1:3.6.7-1 @/ovirt-release36 ovirt-release40.noarch 4.0.3-1 @/ovirt-release40 ovirt-setup-lib.noarch 1.0.1-1.el7.centos @ovirt-3.6 ovirt-vmconsole.noarch 1.0.2-1.el7.centos @ovirt-3.6 ovirt-vmconsole-proxy.noarch 1.0.2-1.el7.centos @ovirt-3.6 Is it safe to assume that the packages just aren't happy and a backup/restore is the best option at this point? Or is this still salvageable? 2016-09-06 15:12:14,682 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 12489166-d1ea-4bcc-8e96-526a1826c506 Type: VDS 2016-09-06 15:12:14,682 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Before Installation host 12489166-d1ea-4bcc-8e96-526a1826c506, srv3.xxx.xxx.xxx 2016-09-06 15:12:14,715 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] START, SetVdsStatusVDSCommand(HostName = srv3.xxx.xxx.xxx, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='12489166-d1ea-4bcc-8e96-526a1826c506', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23376707 2016-09-06 15:12:14,719 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] FINISH, SetVdsStatusVDSCommand, log id: 23376707 2016-09-06 15:12:14,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56, Call Stack: null, Custom Event ID: -1, Message: Installing Host srv3.xxx.xxx.xxx. Connected to host <host ip> with SSH key fingerprint: SHA256:<redacted>. 2016-09-06 15:12:14,824 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Installation of <host ip>. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2016-09-06 15:12:14,824 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh 2016-09-06 15:12:14,852 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] SSH execute 'root@<host ip>' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2016-09-06 15:12:15,125 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] SSH error running command root@<host ip>:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command returned failure code 1 during SSH session 'root@<host ip>' 2016-09-06 15:12:15,121 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [2bfaf56] Error during deploy dialog: java.io.IOException: Unexpected connection termination at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:387) [otopi.jar:] at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:404) [otopi.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase._threadMain(VdsDeployBase.java:304) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.access$800(VdsDeployBase.java:45) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase$12.run(VdsDeployBase.java:386) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-06 15:12:15,126 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Exception: java.io.IOException: Command returned failure code 1 during SSH session 'root@<host ip>' at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) [uutils.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-06 15:12:15,127 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Error during host <host ip> install: java.io.IOException: Command returned failure code 1 during SSH session 'root@<host ip>' at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) [uutils.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-06 15:12:15,156 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host srv3.xxx.xxx.xxx. Command returned failure code 1 during SSH session 'root@<host ip>'. 2016-09-06 15:12:15,156 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Error during host <host ip> install, prefering first exception: Unexpected connection termination 2016-09-06 15:12:15,156 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Host installation failed for host '12489166-d1ea-4bcc-8e96-526a1826c506', 'srv3.xxx.xxx.xxx': Unexpected connection termination 2016-09-06 15:12:15,180 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] START, SetVdsStatusVDSCommand(HostName = srv3.xxx.xxx.xxx, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='12489166-d1ea-4bcc-8e96-526a1826c506', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 523cdcff 2016-09-06 15:12:15,184 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] FINISH, SetVdsStatusVDSCommand, log id: 523cdcff 2016-09-06 15:12:15,190 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Correlation ID: 2bfaf56, Call Stack: null, Custom Event ID: -1, Message: Host srv3.xxx.xxx.xxx installation failed. Unexpected connection termination. 2016-09-06 15:12:15,190 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-32) [2bfaf56] Lock freed to object 'EngineLock:{exclusiveLocks='[12489166-d1ea-4bcc-8e96-526a1826c506=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-06 15:12:15,695 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [2bfaf56] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing power management status on Host srv3.xxx.xxx.xxx using Proxy Host srv2.xxx.xxx.xxx and Fence Agent ipmilan:xxx.xx.xx.xx. 2016-09-06 15:12:15,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-3) [2bfaf56] START, FenceVdsVDSCommand(HostName = srv2.xxx.xxx.xxx, FenceVdsVDSCommandParameters:{runAsync='true', hostId='d1fbefab-c5f7-4c4a-811f-0f4e2cd7d9e3', targetVdsId='12489166-d1ea-4bcc-8e96-526a1826c506', action='STATUS', agent='FenceAgent:{id='7cddafa7-94bd-48fe-b593-fbc3fe24950e', hostId='12489166-d1ea-4bcc-8e96-526a1826c506', order='1', type='ipmilan', ip='xxx.xx.xx.xx', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR delay=10 lanplus=1'}', policy='null'}), log id: 32d678be 2016-09-06 15:12:15,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-3) [2bfaf56] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 32d678be 2016-09-06 15:12:15,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [2bfaf56] Correlation ID: 2bfaf56, Call Stack: null, Custom Event ID: -1, Message: Host srv3.xxx.xxx.xxx configuration was updated by xxx(a)xxx.xxx. On Tue, 6 Sep 2016, at 05:01 PM, Yedidyah Bar David wrote: > On Tue, Sep 6, 2016 at 6:08 AM, James <mailinglists(a)mooash.com&gt; wrote: > > Hey, > > > > I moved our standalone engine from a C6 host to a C7 host and now it > > wont install/setup oVirt nodes. It just responds with messages like > > this: > > > > Host xxx.xxx.xxx installation failed. Unexpected connection termination. > > Failed to install Host xxx.xxx.xxx. Command returned failure code 1 > > during SSH session &#39;root(a)xxx.xx.xx.x&#39;. > > > > And from engine.log > > > > 2016-09-06 12:20:08,011 ERROR > > [org.ovirt.engine.core.uutils.ssh.SSHDialog] > > (org.ovirt.thread.pool-8-thread-41) [55397a53] SSH error running command > > root@xxx.xx.xx.x:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp > > -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MY > > 2016-09-06 12:20:08,011 ERROR > > [org.ovirt.engine.core.uutils.ssh.SSHDialog] > > (org.ovirt.thread.pool-8-thread-41) [55397a53] SSH error running command > > root@xxx.xx.xx.x:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp > > -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MY > > TMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar > > --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy > > DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command > > returned failure code 1 during SSH session &#39;root(a)xxx.xx.xx.x&#39; > > 2016-09-06 12:20:08,007 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) > > [55397a53] Error during deploy dialog: java.io.IOException: Unexpected > > connection termination > > at > > org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:387) > > [otopi.jar:] > > at > > org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:404) > > [otopi.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase._threadMain(VdsDeployBase.java:304) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.access$800(VdsDeployBase.java:45) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase$12.run(VdsDeployBase.java:386) > > [bll.jar:] > > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] > > > > 2016-09-06 12:20:08,013 ERROR > > [org.ovirt.engine.core.uutils.ssh.SSHDialog] > > (org.ovirt.thread.pool-8-thread-41) [55397a53] Exception: > > java.io.IOException: Command returned failure code 1 during SSH session > > &#39;root(a)xxx.xx.xx.x&#39; > > at > > org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) > > [uutils.jar:] > > at > > org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) > > [uutils.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982) > > [bll.jar:] > > at > > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > > [utils.jar:] > > at > > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > > [utils.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181) > > [bll.jar:] > > at > > org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89) > > [utils.jar:] > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > [rt.jar:1.8.0_101] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > [rt.jar:1.8.0_101] > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [rt.jar:1.8.0_101] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [rt.jar:1.8.0_101] > > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] > > > > 2016-09-06 12:20:08,014 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > > (org.ovirt.thread.pool-8-thread-41) [55397a53] Error during host > > xxx.xx.xx.x install: java.io.IOException: Command returned failure code > > 1 during SSH session &#39;root(a)xxx.xx.xx.x&#39; > > at > > org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) > > [uutils.jar:] > > at > > org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) > > [uutils.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:189) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1215) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1359) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1982) > > [bll.jar:] > > at > > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > > [utils.jar:] > > at > > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > > [utils.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1396) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:378) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:207) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:172) > > [bll.jar:] > > at > > org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:181) > > [bll.jar:] > > at > > org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:89) > > [utils.jar:] > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > [rt.jar:1.8.0_101] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > [rt.jar:1.8.0_101] > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [rt.jar:1.8.0_101] > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [rt.jar:1.8.0_101] > > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] > > > > I've confirmed connectivity from the engine to the hosts themselves and > > it can SSH fine, even running the above command manually works fine. > > > > To confirm, I migrated the oVirt engine by taking a backup, shutting > > down the old host, changing the hostname/IP of the new host to be the > > same as the old and restoring the backup. This is happening on oVirt 3.6 > > and is currently stopping us from being able to upgrade to oVirt 4.0 > > since we cannot convert one of our clusters to 3.6 compatibility mode > > (its currently on 3.5) without migrating the VM hosts. It happens on a > > fresh install or a reinstall, so not sure whats happening. > > Can you please check/attach more of engine.log from before the error? > Start > with the line having 'Before Installation'. Thanks. > -- > Didi