On Oct 26, 2016, at 8:53 PM, Thing <thing.thing@gmail.com> wrote: =20 from the engine log I see, =20 =3D=3D=3D=3D=3D=3D=3D=3D 2016-10-27 13:45:24,595 INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] START, = GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78 2016-10-27 13:46:15,780 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler3) [] No up server in cluster 2016-10-27 13:47:12,326 INFO = [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] = (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster = 'Default' for snapshot monitoring 2016-10-27 13:47:12,329 INFO = [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] = (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster = 'Default' for snapshot configurations monitoring 2016-10-27 13:47:15,783 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler4) [27554ab4] No up server in cluster 2016-10-27 13:48:15,785 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler4) [27554ab4] No up server in cluster 2016-10-27 13:48:24,879 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: = null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: = Message timeout which can be caused by communication issues 2016-10-27 13:48:24,879 ERROR = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] Command = 'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues 2016-10-27 13:48:24,879 INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] FINISH, = GetGlusterHostUUIDVDSCommand, log id: 51f83f78 2016-10-27 13:48:24,879 ERROR = [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) = [3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' = failed: EngineException: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues (Failed with error VDS_NETWORK_ERROR and = code 5022) 2016-10-27 13:48:24,887 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call = Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer =
--Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii This is pointing to network connectivity issues You can look at logs under /var/logs/ovirt-* Check your dns and make sure the hosts can connect to each other. You = can restart the host, mark it as restarted in the manager and then = activate it. Worst case, you can remove it using hosted-engine commands Tom list from server glusterp1 on Cluster Default.
2016-10-27 13:48:24,908 INFO = [org.ovirt.engine.core.bll.HandleVdsVersionCommand] = (DefaultQuartzScheduler2) [6083469d] Running command: = HandleVdsVersionCommand internal: true. Entities affected : ID: = 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS 2016-10-27 13:48:24,959 INFO = [org.ovirt.engine.core.vdsbroker.VdsManager] = (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, = vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1', = vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, = error: VDSGenericException: VDSNetworkException: Message timeout which = can be caused by communication issues 2016-10-27 13:48:25,003 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, = Call Stack: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues at = org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe= turnValue(BrokerCommandBase.java:188) at = org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce= edProxyReturnValue(AbstractGlusterBrokerCommand.java:122) at = org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu= teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19) at = org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma= nd(VdsBrokerCommand.java:110) at = org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa= se.java:73) at = org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) at = org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana= ger.java:451) at = org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro= ntendImpl.java:33) at = org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)=
at =
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand= (StorageHandlingCommandBase.java:657)
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo= mmand.java:397)
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom= mand.java:149)
at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas= e.java:1305)
at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm= andBase.java:1447)
at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20= 75)
at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre= ssed(TransactionSupport.java:166)
at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(= TransactionSupport.java:105)
at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) at =
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
at =
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source) at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat= ion(ManagedReferenceMethodInterceptor.java:52)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29= 9BindingsInterceptor.java:70)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29= 9BindingsInterceptor.java:80)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi= ndingsInterceptor.java:93)
at =
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv= ocation(UserInterceptorFactory.java:63)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro= cessInvocation(ExecutionTimeInterceptor.java:43)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437)
at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo= ke(AbstractEJBRequestScopeActivationInterceptor.java:73)
at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati= on(EjbRequestScopeActivationInterceptor.java:83)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(= ConcurrentContextInterceptor.java:45)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept= or.java:21)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)
at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc= essInvocation(ComponentDispatcherInterceptor.java:52)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio= nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc= eptor.java:53)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2= 63)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j= ava:243)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept= or.processInvocation(CurrentInvocationContextInterceptor.java:41)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI= nvocation(WaitTimeInterceptor.java:43)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se= curityContextInterceptor.java:100)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc= essInvocation(ShutDownInterceptorFactory.java:64)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat= ion(LoggingInterceptor.java:66)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na= mespaceContextInterceptor.java:50)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte= xtClassLoaderInterceptor.java:64)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur= ityManager.java:636)
at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe= ckingInterceptor.java:61)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P= rivilegedWithCombinerInterceptor.java:80)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)
at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript= ion.java:185)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)
at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan= dler.java:73)
at =
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc= tion(Unknown Source)
at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source) at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at =
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav= a:436)
at =
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri= seBeanProxyMethodHandler.java:127)
at =
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa= rgetBeanInstance.java:56)
at =
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav= a:100)
at =
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac= kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk= nown Source)
at =
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav= a:308)
at =
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni= toring.java:93)
at =
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at =
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:= 77)
at =
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213) at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1142)
at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617)
at java.lang.Thread.run(Thread.java:745) , Custom Event ID: -1, Message: Host glusterp1 is non responsive. 2016-10-27 13:48:25,019 INFO =
2016-10-27 13:48:25,054 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD= S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}' 2016-10-27 13:48:25,069 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: = VdsNotRespondingTreatmentCommand internal: true. Entities affected : = ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS 2016-10-27 13:48:25,079 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: = SshSoftFencingCommand internal: true. Entities affected : ID: = 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS 2016-10-27 13:48:25,081 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing = session on host '192.168.1.31' 2016-10-27 13:48:25,189 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft = Fencing command on host '192.168.1.31' 2016-10-27 13:48:27,914 INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to /192.168.1.31 <http://192.168.1.31/> 2016-10-27 13:48:29,379 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] = (DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D= glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', = vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: = 46d1a470 2016-10-27 13:48:30,383 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] = (DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: = 46d1a470 2016-10-27 13:48:30,449 INFO = [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] = (DefaultQuartzScheduler7) [a8ef973] Running command: = HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities = affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS 2016-10-27 13:48:30,464 INFO = [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) = [1c434304] Running command: InitVdsOnUpCommand internal: true. Entities = affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool 2016-10-27 13:48:30,469 INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler7) [1c434304] START, = GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8 2016-10-27 13:48:37,296 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD= S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}' 2016-10-27 13:48:37,328 INFO = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, = Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom = Event ID: -1, Message: Host glusterp1 is rebooting. 2016-10-27 13:48:37,336 WARN = [org.ovirt.engine.core.bll.lock.InMemoryLockManager] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release = exclusive lock which does not exist, lock key: = '260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE' 2016-10-27 13:48:37,336 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD= S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}' 2016-10-27 13:49:15,787 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler8) [beed7f3] No up server in cluster 2016-10-27 13:50:15,790 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler1) [2d374249] No up server in cluster =3D=3D=3D=3D=3D=3D=3D =20 =20 =20 On 27 October 2016 at 13:45, Thing <thing.thing@gmail.com = <mailto:thing.thing@gmail.com>> wrote: While trying to figure out how to deploy storage I put 1 host into =
[org.ovirt.engine.core.bll.VdsEventListener] = (org.ovirt.thread.pool-8-thread-29) [3877c2ef] = ResourceManager::vdsNotResponding entered for Host = '260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31' maintenance mode, trying to re-activate it and its failed.
=20 It seems to be stuck as neither in activated nor maintenance, so how = would I go about fixing this? =20 So what log(s) would this be written to? =20 =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">This is pointing to network connectivity = issues</div><div class=3D""><br class=3D""></div><div class=3D"">You can = look at logs under /var/logs/ovirt-*</div><div class=3D""><br = class=3D""></div><div class=3D"">Check your dns and make sure the hosts = can connect to each other. You can restart the host, mark it as = restarted in the manager and then activate it.</div><div class=3D""><br = class=3D""></div><div class=3D"">Worst case, you can remove it using = hosted-engine commands</div><div class=3D""><br class=3D""></div><div = class=3D"">Tom</div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Oct 26, 2016, at 8:53 PM, Thing <<a = href=3D"mailto:thing.thing@gmail.com" = class=3D"">thing.thing@gmail.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">from the engine log I see,<br class=3D""><br = class=3D"">=3D=3D=3D=3D=3D=3D=3D=3D<br class=3D"">2016-10-27 = 13:45:24,595 INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] START, = GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78<br = class=3D"">2016-10-27 13:46:15,780 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler3) [] No up server in cluster<br = class=3D"">2016-10-27 13:47:12,326 INFO = [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] = (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster = 'Default' for snapshot monitoring<br class=3D"">2016-10-27 13:47:12,329 = INFO [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] = (DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster = 'Default' for snapshot configurations monitoring<br class=3D"">2016-10-27 = 13:47:15,783 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br = class=3D"">2016-10-27 13:48:15,785 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br = class=3D"">2016-10-27 13:48:24,879 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: = null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: = Message timeout which can be caused by communication issues<br = class=3D"">2016-10-27 13:48:24,879 ERROR = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] Command = 'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues<br class=3D"">2016-10-27 13:48:24,879 = INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler2) [3877c2ef] FINISH, = GetGlusterHostUUIDVDSCommand, log id: 51f83f78<br class=3D"">2016-10-27 = 13:48:24,879 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] = (DefaultQuartzScheduler2) [3877c2ef] Command = 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed: EngineException: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues (Failed with error VDS_NETWORK_ERROR and = code 5022)<br class=3D"">2016-10-27 13:48:24,887 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call = Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer = list from server glusterp1 on Cluster Default.<br class=3D"">2016-10-27 = 13:48:24,908 INFO = [org.ovirt.engine.core.bll.HandleVdsVersionCommand] = (DefaultQuartzScheduler2) [6083469d] Running command: = HandleVdsVersionCommand internal: true. Entities affected : ID: = 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 = 13:48:24,959 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] = (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, = vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1', = vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, = error: VDSGenericException: VDSNetworkException: Message timeout which = can be caused by communication issues<br class=3D"">2016-10-27 = 13:48:25,003 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, = Call Stack: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Message timeout which can be = caused by communication issues<br class=3D""> at = org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe= turnValue(BrokerCommandBase.java:188)<br class=3D""> = at = org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce= edProxyReturnValue(AbstractGlusterBrokerCommand.java:122)<br = class=3D""> at = org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu= teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)<br = class=3D""> at = org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma= nd(VdsBrokerCommand.java:110)<br class=3D""> at = org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa= se.java:73)<br class=3D""> at = org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)<b= r class=3D""> at = org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana= ger.java:451)<br class=3D""> at = org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro= ntendImpl.java:33)<br class=3D""> at = org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)= <br class=3D""> at = org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand= (StorageHandlingCommandBase.java:657)<br class=3D""> = at = org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo= mmand.java:397)<br class=3D""> at = org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom= mand.java:149)<br class=3D""> at = org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas= e.java:1305)<br class=3D""> at = org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm= andBase.java:1447)<br class=3D""> at = org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20= 75)<br class=3D""> at = org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre= ssed(TransactionSupport.java:166)<br class=3D""> at = org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(= TransactionSupport.java:105)<br class=3D""> at = org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)<br = class=3D""> at = org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)<= br class=3D""> at = org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)<br = class=3D""> at = org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)<br = class=3D""> at = org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)<br = class=3D""> at = sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)<br = class=3D""> at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)<br class=3D""> at = java.lang.reflect.Method.invoke(Method.java:498)<br = class=3D""> at = org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat= ion(ManagedReferenceMethodInterceptor.java:52)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437)<br class=3D""> at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29= 9BindingsInterceptor.java:70)<br class=3D""> at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29= 9BindingsInterceptor.java:80)<br class=3D""> at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi= ndingsInterceptor.java:93)<br class=3D""> at = org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv= ocation(UserInterceptorFactory.java:63)<br class=3D""> = at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro= cessInvocation(ExecutionTimeInterceptor.java:43)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437)<br class=3D""> at = org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo= ke(AbstractEJBRequestScopeActivationInterceptor.java:73)<br = class=3D""> at = org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati= on(EjbRequestScopeActivationInterceptor.java:83)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(= ConcurrentContextInterceptor.java:45)<br class=3D""> = at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept= or.java:21)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)<br class=3D""> at = org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc= essInvocation(ComponentDispatcherInterceptor.java:52)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio= nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc= eptor.java:53)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2= 63)<br class=3D""> at = org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)<= br class=3D""> at = org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j= ava:243)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept= or.processInvocation(CurrentInvocationContextInterceptor.java:41)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI= nvocation(WaitTimeInterceptor.java:43)<br class=3D""> = at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se= curityContextInterceptor.java:100)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc= essInvocation(ShutDownInterceptorFactory.java:64)<br = class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat= ion(LoggingInterceptor.java:66)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na= mespaceContextInterceptor.java:50)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte= xtClassLoaderInterceptor.java:64)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b= r class=3D""> at = org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur= ityManager.java:636)<br class=3D""> at = org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe= ckingInterceptor.java:61)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b= r class=3D""> at = org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P= rivilegedWithCombinerInterceptor.java:80)<br class=3D""> = at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)<br class=3D""> at = org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)<br= class=3D""> at = org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript= ion.java:185)<br class=3D""> at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0)<br class=3D""> at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61)<br class=3D""> at = org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan= dler.java:73)<br class=3D""> at = org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc= tion(Unknown Source)<br class=3D""> at = sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)<br = class=3D""> at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)<br class=3D""> at = java.lang.reflect.Method.invoke(Method.java:498)<br = class=3D""> at = org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav= a:436)<br class=3D""> at = org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri= seBeanProxyMethodHandler.java:127)<br class=3D""> at = org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa= rgetBeanInstance.java:56)<br class=3D""> at = org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav= a:100)<br class=3D""> at = org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac= kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk= nown Source)<br class=3D""> at = org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav= a:308)<br class=3D""> at = org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni= toring.java:93)<br class=3D""> at = org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)<br= class=3D""> at = sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)<br = class=3D""> at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)<br class=3D""> at = java.lang.reflect.Method.invoke(Method.java:498)<br = class=3D""> at = org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:= 77)<br class=3D""> at = org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)<b= r class=3D""> at = org.quartz.core.JobRunShell.run(JobRunShell.java:213)<br = class=3D""> at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br= class=3D""> at = java.util.concurrent.FutureTask.run(FutureTask.java:266)<br = class=3D""> at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1142)<br class=3D""> at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617)<br class=3D""> at = java.lang.Thread.run(Thread.java:745)<br class=3D"">, Custom Event ID: = -1, Message: Host glusterp1 is non responsive.<br class=3D"">2016-10-27 = 13:48:25,019 INFO [org.ovirt.engine.core.bll.VdsEventListener] = (org.ovirt.thread.pool-8-thread-29) [3877c2ef] = ResourceManager::vdsNotResponding entered for Host = '260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'<br = class=3D"">2016-10-27 13:48:25,054 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<= ;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:25,069 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: = VdsNotRespondingTreatmentCommand internal: true. Entities affected = : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br = class=3D"">2016-10-27 13:48:25,079 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: = SshSoftFencingCommand internal: true. Entities affected : ID: = 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 = 13:48:25,081 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing = session on host '192.168.1.31'<br class=3D"">2016-10-27 13:48:25,189 = INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft = Fencing command on host '192.168.1.31'<br class=3D"">2016-10-27 = 13:48:27,914 INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to /<a href=3D"http://192.168.1.31/" = class=3D"">192.168.1.31</a><br class=3D"">2016-10-27 13:48:29,379 = INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] = (DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D= glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', = vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: = 46d1a470<br class=3D"">2016-10-27 13:48:30,383 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] = (DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: = 46d1a470<br class=3D"">2016-10-27 13:48:30,449 INFO = [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] = (DefaultQuartzScheduler7) [a8ef973] Running command: = HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities = affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br = class=3D"">2016-10-27 13:48:30,464 INFO = [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) = [1c434304] Running command: InitVdsOnUpCommand internal: true. Entities = affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: = StoragePool<br class=3D"">2016-10-27 13:48:30,469 INFO = [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] = (DefaultQuartzScheduler7) [1c434304] START, = GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8<br = class=3D"">2016-10-27 13:48:37,296 INFO = [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<= ;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:37,328 INFO = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, = Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom = Event ID: -1, Message: Host glusterp1 is rebooting.<br = class=3D"">2016-10-27 13:48:37,336 WARN = [org.ovirt.engine.core.bll.lock.InMemoryLockManager] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release = exclusive lock which does not exist, lock key: = '260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'<br class=3D"">2016-10-27 = 13:48:37,336 INFO = [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] = (org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<= ;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', = sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:49:15,787 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler8) [beed7f3] No up server in cluster<br = class=3D"">2016-10-27 13:50:15,790 INFO = [org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] = (DefaultQuartzScheduler1) [2d374249] No up server in cluster<br = class=3D"">=3D=3D=3D=3D=3D=3D=3D<br class=3D""><br class=3D""><br = class=3D""></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On 27 October 2016 at 13:45, Thing <span dir=3D"ltr"= class=3D""><<a href=3D"mailto:thing.thing@gmail.com" target=3D"_blank"= class=3D"">thing.thing@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D""><div class=3D""><div class=3D"">While trying to figure out = how to deploy storage I put 1 host into maintenance mode, trying to = re-activate it and its failed.<br class=3D""></div><br class=3D""></div>It= seems to be stuck as neither in activated nor maintenance, so how would = I go about fixing this?<br class=3D""><br class=3D"">So what log(s) = would this be written to? <br class=3D""></div> </blockquote></div><br class=3D""></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA--