[ovirt-users] Additional hosted-engine host deployment failed due to message timeout

Tue May 3 07:29:29 UTC 2016

On Tue, May 3, 2016 at 5:05 AM, Wee Sritippho <wee.s at forest.go.th> wrote:
> Hi,
>
> I'm making a fresh hosted-engine installation on 3 hosts. First 2 hosts
> succeeded, but the 3rd one stuck at termination state "Installing Host
> hosted_engine_3. Stage: Termination.", then, 3 minutes later, "VDSM
> hosted_engine_3 command failed: Message timeout which can be caused by
> communication issues". Currently, the 3rd host status in web UI is stuck at
> "Installing".
>
> How can I proceed? Could I just run this script in the 3rd host, and run
> "hosted-engine --deploy" again?
>
> log files: https://app.box.com/s/a5typfe6cbozs9uo9osg68gtmq8793t6

The error is here:
2016-05-03 08:43:49,760 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-9) [454e2dc8] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine_3
command failed: Message timeout which can be caused by communication
issues
2016-05-03 08:43:49,760 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
(org.ovirt.thread.pool-8-thread-9) [454e2dc8] Command
'PollVDSCommand(HostName = hosted_engine_3,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='ed3526a1-6a03-44a7-a9a5-e3b346384ac4'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-05-03 08:43:49,760 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
(org.ovirt.thread.pool-8-thread-9) [454e2dc8] Error:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-05-03 08:43:49,760 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
(org.ovirt.thread.pool-8-thread-9) [454e2dc8] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
[vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73)
[vdsbroker.jar:]
at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:30)
[bll.jar:]
at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:423)
[bll.jar:]
at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:256)
[bll.jar:]

This is something that happens after the engine called setupNetworks
and, due to some side effects, it loose the communication with the
host.

The issue is tracked here:
https://bugzilla.redhat.com/1323465
we are going to solve for 3.6.6

As Knarra said, you can simply try redeploying it and it could work:
this issue is not fully systematic.
In order to retry you need to remove your host from the engine and in
order to do that you probably need to move it to maintenance mode.

> [me at host03 ~]$ rpm -qa | grep ovirt
> ovirt-release36-007-1.noarch
> ovirt-hosted-engine-setup-1.3.5.0-1.1.el7.noarch
> libgovirt-0.3.3-1.el7_2.1.x86_64
> ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch
> ovirt-setup-lib-1.0.1-1.el7.centos.noarch
> ovirt-vmconsole-1.0.0-1.el7.centos.noarch
> ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
> ovirt-host-deploy-1.4.1-1.el7.centos.noarch
> ovirt-hosted-engine-ha-1.3.5.3-1.1.el7.noarch
>
> [me at host03 ~]$ rpm -qa | grep vdsm
> vdsm-hook-vmfex-dev-4.17.26-1.el7.noarch
> vdsm-xmlrpc-4.17.26-1.el7.noarch
> vdsm-infra-4.17.26-1.el7.noarch
> vdsm-yajsonrpc-4.17.26-1.el7.noarch
> vdsm-python-4.17.26-1.el7.noarch
> vdsm-4.17.26-1.el7.noarch
> vdsm-cli-4.17.26-1.el7.noarch
> vdsm-jsonrpc-4.17.26-1.el7.noarch
>
> [root at engine ~]# rpm -qa | grep ovirt
> ovirt-engine-wildfly-8.2.1-1.el7.x86_64
> ovirt-engine-setup-plugin-ovirt-engine-common-3.6.5.3-1.el7.centos.noarch
> ovirt-vmconsole-1.0.0-1.el7.centos.noarch
> ovirt-engine-cli-3.6.2.0-1.el7.centos.noarch
> ovirt-engine-setup-plugin-ovirt-engine-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-backend-3.6.5.3-1.el7.centos.noarch
> ovirt-iso-uploader-3.6.0-1.el7.centos.noarch
> ovirt-engine-extensions-api-impl-3.6.5.3-1.el7.centos.noarch
> ovirt-host-deploy-1.4.1-1.el7.centos.noarch
> ovirt-release36-007-1.noarch
> ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch
> ovirt-image-uploader-3.6.0-1.el7.centos.noarch
> ovirt-engine-extension-aaa-jdbc-1.0.6-1.el7.noarch
> ovirt-setup-lib-1.0.1-1.el7.centos.noarch
> ovirt-host-deploy-java-1.4.1-1.el7.centos.noarch
> ovirt-engine-setup-base-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-setup-plugin-websocket-proxy-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-tools-backup-3.6.5.3-1.el7.centos.noarch
> ovirt-vmconsole-proxy-1.0.0-1.el7.centos.noarch
> ovirt-engine-vmconsole-proxy-helper-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-setup-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-webadmin-portal-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-tools-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-restapi-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-wildfly-overlay-8.0.5-1.el7.noarch
> ovirt-engine-lib-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-websocket-proxy-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-setup-plugin-vmconsole-proxy-helper-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-userportal-3.6.5.3-1.el7.centos.noarch
> ovirt-engine-dbscripts-3.6.5.3-1.el7.centos.noarch
>
> Thanks,
>
> --
> Wee
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>