[ovirt-users] failed to activate a (gluster) host. ovirt 4.0.4

Thing thing.thing at gmail.com
Thu Oct 27 00:53:08 UTC 2016 

from the engine log I see,

========
2016-10-27 13:45:24,595 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] START,
GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78
2016-10-27 13:46:15,780 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler3) [] No up server in cluster
2016-10-27 13:47:12,326 INFO
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob]
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster
'Default' for snapshot monitoring
2016-10-27 13:47:12,329 INFO
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob]
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster
'Default' for snapshot configurations monitoring
2016-10-27 13:47:15,783 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:15,785 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: Message
timeout which can be caused by communication issues
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] Command
'GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-10-27 13:48:24,879 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] FINISH, GetGlusterHostUUIDVDSCommand,
log id: 51f83f78
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2)
[3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)
2016-10-27 13:48:24,887 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call Stack:
null, Custom Event ID: -1, Message: Failed to fetch gluster peer list from
server glusterp1 on Cluster Default.
2016-10-27 13:48:24,908 INFO
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(DefaultQuartzScheduler2) [6083469d] Running command:
HandleVdsVersionCommand internal: true. Entities affected :  ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:24,959 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager]
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond,
vds_id='260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name='glusterp1',
vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-10-27 13:48:25,003 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, Call
Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
    at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
    at
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proceedProxyReturnValue(AbstractGlusterBrokerCommand.java:122)
    at
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.executeVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)
    at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
    at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)
    at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
    at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451)
    at
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
    at
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)
    at
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657)
    at
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCommand.java:397)
    at
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCommand.java:149)
    at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305)
    at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447)
    at
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075)
    at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
    at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
    at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
    at
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
    at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)
    at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
    at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
    at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
    at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
    at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
    at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
    at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
    at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
    at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
    at
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
    at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
    at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
    at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
    at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
    at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
Source)
    at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:436)
    at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
    at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
    at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
    at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source)
    at
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.java:308)
    at
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:93)
    at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
    at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77)
    at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
, Custom Event ID: -1, Message: Host glusterp1 is non responsive.
2016-10-27 13:48:25,019 INFO  [org.ovirt.engine.core.bll.VdsEventListener]
(org.ovirt.thread.pool-8-thread-29) [3877c2ef]
ResourceManager::vdsNotResponding entered for Host
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'
2016-10-27 13:48:25,054 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:48:25,069 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command:
VdsNotRespondingTreatmentCommand internal: true. Entities affected :  ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,079 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command:
SshSoftFencingCommand internal: true. Entities affected :  ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,081 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing
session on host '192.168.1.31'
2016-10-27 13:48:25,189 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft Fencing
command on host '192.168.1.31'
2016-10-27 13:48:27,914 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to /192.168.1.31
2016-10-27 13:48:29,379 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5',
vds='Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id:
46d1a470
2016-10-27 13:48:30,383 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id:
46d1a470
2016-10-27 13:48:30,449 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(DefaultQuartzScheduler7) [a8ef973] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected
:  ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:30,464 INFO
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7)
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities
affected :  ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool
2016-10-27 13:48:30,469 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler7) [1c434304] START,
GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8
2016-10-27 13:48:37,296 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:48:37,328 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659,
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom
Event ID: -1, Message: Host glusterp1 is rebooting.
2016-10-27 13:48:37,336 WARN
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release exclusive
lock which does not exist, lock key:
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'
2016-10-27 13:48:37,336 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:49:15,787 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster
2016-10-27 13:50:15,790 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler1) [2d374249] No up server in cluster
=======



On 27 October 2016 at 13:45, Thing <thing.thing at gmail.com> wrote:

> While trying to figure out how to deploy storage I put 1 host into
> maintenance mode, trying to re-activate it and its failed.
>
> It seems to be stuck as neither in activated nor maintenance, so how would
> I go about fixing this?
>
> So what log(s) would this be written to?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161027/8f870bd1/attachment-0001.html>

More information about the Users mailing list