[ovirt-devel] Installing host from engine fails (master)

This is a multi-part message in MIME format. --------------080704070103080902010009 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, Engine in master is having problems activating a CentOS 7 host. The cause seems to be related with "missing" plugins: 2015-04-20 00:32:20,955 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) [50ddd3ed] Installation 'h4.imatronix.com': Internal error: Internal error, plugins set(['otopi', 'setup']) are missing We have this RPMS: otopi-1.4.0-0.0.master.20150409182110.git0d2105b.el7.noarch otopi-java-1.4.0-0.0.master.20150409182110.git0d2105b.el7.noarch ovirt-engine-setup-plugin-ovirt-engine-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch ovirt-engine-setup-plugin-allinone-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch ovirt-engine-setup-plugin-websocket-proxy-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch More logs: 2015-04-20 00:32:20,830 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] SSH execute &#39;root(a)h4.imatronix.com&#39; 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2015-04-20 00:32:20,955 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) [50ddd3ed] Installation 'h4.imatronix.com': Internal error: Internal error, plugins set(['otopi', 'setup']) are missing 2015-04-20 00:32:20,970 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] SSH error running command root@h4.imatronix.com:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command returned failure code 2 during SSH session &#39;root(a)h4.imatronix.com&#39; 2015-04-20 00:32:20,970 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Exception: java.io.IOException: Command returned failure code 2 during SSH session &#39;root(a)h4.imatronix.com&#39; at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:] at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) [uutils.jar:] at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1125) [bll.jar:] at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:165) [bll.jar:] at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:91) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1183) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1327) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1925) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1351) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:346) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:202) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:170) [bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:179) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-04-20 00:32:20,989 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host H4. Internal error: Internal error, plugins set(['otopi', 'setup']) are missing. 2015-04-20 00:32:20,992 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) [50ddd3ed] Error during deploy dialog: java.io.IOException: Unexpected connection termination at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:388) [otopi.jar:] at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:405) [otopi.jar:] at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:828) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:87) [bll.jar:] at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:976) [bll.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-04-20 00:32:20,996 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Error during host h4.imatronix.com install 2015-04-20 00:32:20,996 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Installation 'h4.imatronix.com': Command returned failure code 2 during SSH session &#39;root(a)h4.imatronix.com&#39; 2015-04-20 00:32:21,034 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host H4. Command returned failure code 2 during SSH session &#39;root(a)h4.imatronix.com&#39;. 2015-04-20 00:32:21,034 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Error during host h4.imatronix.com install, prefering first exception: Unexpected connection termination 2015-04-20 00:32:21,035 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Host installation failed for host 'c579682d-693b-4491-9040-f77ae8e923cc', 'H4': Unexpected connection termination 2015-04-20 00:32:21,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] START, SetVdsStatusVDSCommand(HostName = H4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='c579682d-693b-4491-9040-f77ae8e923cc', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bd1ce90 2015-04-20 00:32:21,079 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] FINISH, SetVdsStatusVDSCommand, log id: 2bd1ce90 2015-04-20 00:32:21,112 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Host H4 installation failed. Unexpected connection termination. 2015-04-20 00:32:21,117 INFO [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Lock freed to object 'EngineLock:{exclusiveLocks='[c579682d-693b-4491-9040-f77ae8e923cc=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' --------------080704070103080902010009 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi,<br> <br> Engine in master is having problems activating a CentOS 7 host.<br> The cause seems to be related with "missing" plugins:<br> <blockquote>2015-04-20 00:32:20,955 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) [50ddd3ed] Installation 'h4.imatronix.com': Internal error: Internal error, plugins set(['otopi', 'setup']) are missing<br> </blockquote> We have this RPMS:<br> <blockquote>otopi-1.4.0-0.0.master.20150409182110.git0d2105b.el7.noarch<br> otopi-java-1.4.0-0.0.master.20150409182110.git0d2105b.el7.noarch<br> ovirt-engine-setup-plugin-ovirt-engine-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch<br> ovirt-engine-setup-plugin-allinone-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch<br> ovirt-engine-setup-plugin-ovirt-engine-common-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch<br> ovirt-engine-setup-plugin-websocket-proxy-3.6.0-0.0.master.20150419172236.gitb6adbca.el7.centos.noarch<br> </blockquote> More logs:<br> <br> <blockquote>2015-04-20 00:32:20,830 INFO  [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] SSH execute '<a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" &gt; /dev/null 2&gt;&amp;1; rm -fr \"${MYTMP}\" &gt; /dev/null 2&gt;&amp;1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &amp;&amp;  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True'<br> 2015-04-20 00:32:20,955 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) [50ddd3ed] Installation 'h4.imatronix.com': Internal error: Internal error, plugins set(['otopi', 'setup']) are missing<br> 2015-04-20 00:32:20,970 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] SSH error running command <a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" &gt; /dev/null 2&gt;&amp;1; rm -fr \"${MYTMP}\" &gt; /dev/null 2&gt;&amp;1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &amp;&amp;  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command returned failure code 2 during SSH session '<a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>'<br> 2015-04-20 00:32:20,970 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Exception: java.io.IOException: Command returned failure code 2 during SSH session '<a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>'<br>         at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:]<br>         at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) [uutils.jar:]<br>         at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1125) [bll.jar:]<br>         at org.ovirt.engine.core.bll.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:165) [bll.jar:]<br>         at org.ovirt.engine.core.bll.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:91) [bll.jar:]<br>         at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1183) [bll.jar:]<br>         at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1327) [bll.jar:]<br>         at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1925) [bll.jar:]<br>         at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]<br>         at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]<br>         at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1351) [bll.jar:]<br>         at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:346) [bll.jar:]<br>         at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:202) [bll.jar:]<br>         at org.ovirt.engine.core.bll.MultipleActionsRunner.runCommands(MultipleActionsRunner.java:170) [bll.jar:]<br>         at org.ovirt.engine.core.bll.MultipleActionsRunner$2.run(MultipleActionsRunner.java:179) [bll.jar:]<br>         at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:]<br>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_79]<br>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_79]<br>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]<br>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]<br>         at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]<br> <br> 2015-04-20 00:32:20,989 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host H4. Internal error: Internal error, plugins set(['otopi', 'setup']) are missing.<br> 2015-04-20 00:32:20,992 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) [50ddd3ed] Error during deploy dialog: java.io.IOException: Unexpected connection termination<br>         at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:388) [otopi.jar:]<br>         at org.ovirt.otopi.dialog.MachineDialogParser.nextEvent(MachineDialogParser.java:405) [otopi.jar:]<br>         at org.ovirt.engine.core.bll.VdsDeploy._threadMain(VdsDeploy.java:828) [bll.jar:]<br>         at org.ovirt.engine.core.bll.VdsDeploy.access$2000(VdsDeploy.java:87) [bll.jar:]<br>         at org.ovirt.engine.core.bll.VdsDeploy$51.run(VdsDeploy.java:976) [bll.jar:]<br>         at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]<br> <br> 2015-04-20 00:32:20,996 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Error during host h4.imatronix.com install<br> 2015-04-20 00:32:20,996 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Installation 'h4.imatronix.com': Command returned failure code 2 during SSH session '<a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>'<br> 2015-04-20 00:32:21,034 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host H4. Command returned failure code 2 during SSH session '<a class="moz-txt-link-abbreviated" href="mailto:root@h4.imatronix.com">root@h4.imatronix.com</a>'.<br> 2015-04-20 00:32:21,034 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Error during host h4.imatronix.com install, prefering first exception: Unexpected connection termination<br> 2015-04-20 00:32:21,035 ERROR [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Host installation failed for host 'c579682d-693b-4491-9040-f77ae8e923cc', 'H4': Unexpected connection termination<br> 2015-04-20 00:32:21,037 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] START, SetVdsStatusVDSCommand(HostName = H4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='c579682d-693b-4491-9040-f77ae8e923cc', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bd1ce90<br> 2015-04-20 00:32:21,079 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] FINISH, SetVdsStatusVDSCommand, log id: 2bd1ce90<br> 2015-04-20 00:32:21,112 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Correlation ID: 50ddd3ed, Call Stack: null, Custom Event ID: -1, Message: Host H4 installation failed. Unexpected connection termination.<br> 2015-04-20 00:32:21,117 INFO  [org.ovirt.engine.core.bll.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-3) [50ddd3ed] Lock freed to object 'EngineLock:{exclusiveLocks='[c579682d-693b-4491-9040-f77ae8e923cc=&lt;VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED&gt;]', sharedLocks='null'}'<br> </blockquote> <br> </body> </html> --------------080704070103080902010009--