[ovirt-users] adding 3 machines as gluster nodes to ovirt 4.0.4

Hi, I have ovirt 4.0.4 running on a centos 7.2 machine. I have 3 identical centos 7.2 machines I want to add as a gluster storage 3 way mirror array. The admin guide doesnt seem to show how to do this? I have setup ssh keys for root access. I have setup a 1TB LUN on each ready for "glusterising". I have tried to build and setup gluster before ahnd and import the ready made setup but this locked up in "activating" and after 36hours never completed (I assume they should move to "up"?) Is there any documentation out there on how? I want these as pure gluster storage only nodes no VMs to go on them. The VMs will go on 2 or 3 new machines I will add later as a next stage. I have tried to add them as hosts (not sure if this is the right method I suspect not), but I get install failures in the engine.log sample, ========= 2016-10-26 11:28:13,899 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-21) [2a413474] SSH error running command root@192.168.1.31:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': Command returned failure code 1 during SSH session &#39;root(a)192.168.1.31&#39; 2016-10-26 11:28:13,899 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (org.ovirt.thread.pool-8-thread-21) [2a413474] Exception: java.io.IOException: Command returned failure code 1 during SSH session ' root(a)192.168.1.31&#39; at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:526) [uutils.jar:] at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:317) [uutils.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:563) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:169) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:90) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-10-26 11:28:13,900 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (org.ovirt.thread.pool-8-thread-21) [2a413474] Error during host 192.168.1.31 install: java.io.IOException: Command returned failure code 1 during SSH session &#39;root(a)192.168.1.31&#39; at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:526) [uutils.jar:] at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:317) [uutils.jar:] at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:563) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:169) [bll.jar:] at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:90) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-10-26 11:28:13,903 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-21) [2a413474] Host installation failed for host '260c0a92-2856-4cd6-a784-01ac95fc41d5', 'glusterp1': Command returned failure code 1 during SSH session &#39;root(a)192.168.1.31&#39; 2016-10-26 11:28:13,905 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [2a413474] START, SetVdsStatusVDSCommand(HostName = glusterp1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8f2959c 2016-10-26 11:28:13,909 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [2a413474] FINISH, SetVdsStatusVDSCommand, log id: 8f2959c 2016-10-26 11:28:13,917 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [2a413474] Correlation ID: 2a413474, Call Stack: null, Custom Event ID: -1, Message: Host glusterp1 installation failed. Command returned failure code 1 during SSH session ' root(a)192.168.1.31&#39;. 2016-10-26 11:28:13,917 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-8-thread-21) [2a413474] Lock freed to object 'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-10-26 11:29:14,466 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler7) [] Check for available updates is skipped for host 'glusterp2' due to unsupported host status 'InstallFailed' 2016-10-26 11:32:10,506 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [] Attempting to update VMs/Templates Ovf. ========= ssh keys dont work either (so I had to set a password) even though root can ssh via said keys so that is working OK. Either a mistake by me, or a bug. iptables is off, ===== [root@glusterp1 ~]# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@glusterp1 ~]# ===== ideas please? regards Steven