[ovirt-users] failed to activate a (gluster) host. ovirt 4.0.4
Tom Gamull
tgamull at redhat.com
Thu Oct 27 00:55:49 UTC 2016
This is pointing to network connectivity issues
You can look at logs under /var/logs/ovirt-*
Check your dns and make sure the hosts can connect to each other. You can restart the host, mark it as restarted in the manager and then activate it.
Worst case, you can remove it using hosted-engine commands
Tom
> On Oct 26, 2016, at 8:53 PM, Thing <thing.thing at gmail.com> wrote:
>
> from the engine log I see,
>
> ========
> 2016-10-27 13:45:24,595 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler2) [3877c2ef] START, GetGlusterHostUUIDVDSCommand(HostName = glusterp1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78
> 2016-10-27 13:46:15,780 INFO [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler3) [] No up server in cluster
> 2016-10-27 13:47:12,326 INFO [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster 'Default' for snapshot monitoring
> 2016-10-27 13:47:12,329 INFO [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster 'Default' for snapshot configurations monitoring
> 2016-10-27 13:47:15,783 INFO [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler4) [27554ab4] No up server in cluster
> 2016-10-27 13:48:15,785 INFO [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler4) [27554ab4] No up server in cluster
> 2016-10-27 13:48:24,879 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: Message timeout which can be caused by communication issues
> 2016-10-27 13:48:24,879 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler2) [3877c2ef] Command 'GetGlusterHostUUIDVDSCommand(HostName = glusterp1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
> 2016-10-27 13:48:24,879 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler2) [3877c2ef] FINISH, GetGlusterHostUUIDVDSCommand, log id: 51f83f78
> 2016-10-27 13:48:24,879 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues (Failed with error VDS_NETWORK_ERROR and code 5022)
> 2016-10-27 13:48:24,887 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer list from server glusterp1 on Cluster Default.
> 2016-10-27 13:48:24,908 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [6083469d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:24,959 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, vds_id='260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name='glusterp1', vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
> 2016-10-27 13:48:25,003 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
> at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
> at org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proceedProxyReturnValue(AbstractGlusterBrokerCommand.java:122)
> at org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.executeVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)
> at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
> at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)
> at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451)
> at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
> at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)
> at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657)
> at org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCommand.java:397)
> at org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCommand.java:149)
> at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305)
> at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447)
> at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075)
> at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
> at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
> at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
> at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)
> at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
> at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
> at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
> at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:436)
> at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
> at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
> at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
> at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source)
> at org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.java:308)
> at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:93)
> at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77)
> at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> , Custom Event ID: -1, Message: Host glusterp1 is non responsive.
> 2016-10-27 13:48:25,019 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-8-thread-29) [3877c2ef] ResourceManager::vdsNotResponding entered for Host '260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'
> 2016-10-27 13:48:25,054 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object 'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}'
> 2016-10-27 13:48:25,069 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:25,079 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:25,081 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing session on host '192.168.1.31'
> 2016-10-27 13:48:25,189 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft Fencing command on host '192.168.1.31'
> 2016-10-27 13:48:27,914 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /192.168.1.31 <http://192.168.1.31/>
> 2016-10-27 13:48:29,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName = glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5', vds='Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: 46d1a470
> 2016-10-27 13:48:30,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: 46d1a470
> 2016-10-27 13:48:30,449 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [a8ef973] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:30,464 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1c434304] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool
> 2016-10-27 13:48:30,469 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] (DefaultQuartzScheduler7) [1c434304] START, GetGlusterHostUUIDVDSCommand(HostName = glusterp1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8
> 2016-10-27 13:48:37,296 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object 'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}'
> 2016-10-27 13:48:37,328 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom Event ID: -1, Message: Host glusterp1 is rebooting.
> 2016-10-27 13:48:37,336 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release exclusive lock which does not exist, lock key: '260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'
> 2016-10-27 13:48:37,336 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object 'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}'
> 2016-10-27 13:49:15,787 INFO [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler8) [beed7f3] No up server in cluster
> 2016-10-27 13:50:15,790 INFO [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] (DefaultQuartzScheduler1) [2d374249] No up server in cluster
> =======
>
>
>
> On 27 October 2016 at 13:45, Thing <thing.thing at gmail.com <mailto:thing.thing at gmail.com>> wrote:
> While trying to figure out how to deploy storage I put 1 host into maintenance mode, trying to re-activate it and its failed.
>
> It seems to be stuck as neither in activated nor maintenance, so how would I go about fixing this?
>
> So what log(s) would this be written to?
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161026/054ddfea/attachment-0001.html>
More information about the Users
mailing list