6:45 p.m.
While trying to figure out how to deploy storage I put 1 host into
maintenance mode, trying to re-activate it and its failed.
It seems to be stuck as neither in activated nor maintenance, so how would
I go about fixing this?
So what log(s) would this be written to?
6:53 p.m.
from the engine log I see,
========
2016-10-27 13:45:24,595 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] START,
GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78
2016-10-27 13:46:15,780 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler3) [] No up server in cluster
2016-10-27 13:47:12,326 INFO
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob]
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster
'Default' for snapshot monitoring
2016-10-27 13:47:12,329 INFO
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob]
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster
'Default' for snapshot configurations monitoring
2016-10-27 13:47:15,783 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:15,785 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: Message
timeout which can be caused by communication issues
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] Command
'GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-10-27 13:48:24,879 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler2) [3877c2ef] FINISH, GetGlusterHostUUIDVDSCommand,
log id: 51f83f78
2016-10-27 13:48:24,879 ERROR
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2)
[3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)
2016-10-27 13:48:24,887 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call Stack:
null, Custom Event ID: -1, Message: Failed to fetch gluster peer list from
server glusterp1 on Cluster Default.
2016-10-27 13:48:24,908 INFO
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(DefaultQuartzScheduler2) [6083469d] Running command:
HandleVdsVersionCommand internal: true. Entities affected : ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:24,959 INFO [org.ovirt.engine.core.vdsbroker.VdsManager]
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond,
vds_id='260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name='glusterp1',
vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2016-10-27 13:48:25,003 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, Call
Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188)
at
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proceedProxyReturnValue(AbstractGlusterBrokerCommand.java:122)
at
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.executeVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73)
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451)
at
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
at
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)
at
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657)
at
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCommand.java:397)
at
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCommand.java:149)
at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305)
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447)
at
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
at
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
Source)
at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:436)
at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source)
at
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.java:308)
at
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:93)
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77)
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
, Custom Event ID: -1, Message: Host glusterp1 is non responsive.
2016-10-27 13:48:25,019 INFO [org.ovirt.engine.core.bll.VdsEventListener]
(org.ovirt.thread.pool-8-thread-29) [3877c2ef]
ResourceManager::vdsNotResponding entered for Host
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'
2016-10-27 13:48:25,054 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:48:25,069 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command:
VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,079 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command:
SshSoftFencingCommand internal: true. Entities affected : ID:
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,081 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing
session on host '192.168.1.31'
2016-10-27 13:48:25,189 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft Fencing
command on host '192.168.1.31'
2016-10-27 13:48:27,914 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to /192.168.1.31
2016-10-27 13:48:29,379 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5',
vds='Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id:
46d1a470
2016-10-27 13:48:30,383 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id:
46d1a470
2016-10-27 13:48:30,449 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(DefaultQuartzScheduler7) [a8ef973] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected
: ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:30,464 INFO
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7)
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities
affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool
2016-10-27 13:48:30,469 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand]
(DefaultQuartzScheduler7) [1c434304] START,
GetGlusterHostUUIDVDSCommand(HostName = glusterp1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8
2016-10-27 13:48:37,296 INFO
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:48:37,328 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659,
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom
Event ID: -1, Message: Host glusterp1 is rebooting.
2016-10-27 13:48:37,336 WARN
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release exclusive
lock which does not exist, lock key:
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'
2016-10-27 13:48:37,336 INFO
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand]
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object
'EngineLock:{exclusiveLocks='[260c0a92-2856-4cd6-a784-01ac95fc41d5=<VDS_FENCE,
POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]',
sharedLocks='null'}'
2016-10-27 13:49:15,787 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster
2016-10-27 13:50:15,790 INFO
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService]
(DefaultQuartzScheduler1) [2d374249] No up server in cluster
=======
On 27 October 2016 at 13:45, Thing <thing.thing(a)gmail.com> wrote:
While trying to figure out how to deploy storage I put 1 host into
maintenance mode, trying to re-activate it and its failed.
It seems to be stuck as neither in activated nor maintenance, so how would
I go about fixing this?
So what log(s) would this be written to?
6:55 p.m.
--Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
This is pointing to network connectivity issues
You can look at logs under /var/logs/ovirt-*
Check your dns and make sure the hosts can connect to each other. You =
can restart the host, mark it as restarted in the manager and then =
activate it.
Worst case, you can remove it using hosted-engine commands
Tom
On Oct 26, 2016, at 8:53 PM, Thing <thing.thing(a)gmail.com>
wrote:
=20
from the engine log I see,
=20
=3D=3D=3D=3D=3D=3D=3D=3D
2016-10-27 13:45:24,595 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78
2016-10-27 13:46:15,780 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler3) [] No up server in cluster
2016-10-27 13:47:12,326 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot monitoring
2016-10-27 13:47:12,329 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot configurations monitoring
2016-10-27 13:47:15,783 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:15,785 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: =
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: =
Message timeout which can be caused by communication issues
2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues
2016-10-27 13:48:24,879 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] FINISH, =
GetGlusterHostUUIDVDSCommand, log id: 51f83f78
2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) =
[3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' =
failed: EngineException: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues (Failed with error VDS_NETWORK_ERROR and =
code 5022)
2016-10-27 13:48:24,887 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call =
Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer =
list from server glusterp1 on Cluster Default.
2016-10-27 13:48:24,908 INFO =
[org.ovirt.engine.core.bll.HandleVdsVersionCommand] =
(DefaultQuartzScheduler2) [6083469d] Running command: =
HandleVdsVersionCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:24,959 INFO =
[org.ovirt.engine.core.vdsbroker.VdsManager] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, =
vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1',
=
vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, =
error: VDSGenericException: VDSNetworkException: Message timeout which =
can be caused by communication issues
2016-10-27 13:48:25,003 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, =
Call Stack: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues
at =
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe=
turnValue(BrokerCommandBase.java:188)
at =
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce=
edProxyReturnValue(AbstractGlusterBrokerCommand.java:122)
at =
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu=
teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)
at =
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma=
nd(VdsBrokerCommand.java:110)
at =
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa=
se.java:73)
at =
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
at =
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana=
ger.java:451)
at =
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro=
ntendImpl.java:33)
at =
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)=
at =
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand=
(StorageHandlingCommandBase.java:657)
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo=
mmand.java:397)
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom=
mand.java:149)
at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1305)
at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1447)
at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
75)
at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre=
ssed(TransactionSupport.java:166)
at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(=
TransactionSupport.java:105)
at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)
at =
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
at =
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
> at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80)
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93)
at =
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro=
cessInvocation(ExecutionTimeInterceptor.java:43)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)
at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73)
at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio=
nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc=
eptor.java:53)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept=
or.processInvocation(CurrentInvocationContextInterceptor.java:41)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI=
nvocation(WaitTimeInterceptor.java:43)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc=
essInvocation(ShutDownInterceptorFactory.java:64)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat=
ion(LoggingInterceptor.java:66)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na=
mespaceContextInterceptor.java:50)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636)
at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185)
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73)
at =
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc=
tion(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
at =
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav=
a:436)
at =
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri=
seBeanProxyMethodHandler.java:127)
at =
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa=
rgetBeanInstance.java:56)
at =
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav=
a:100)
at =
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac=
kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk=
nown Source)
at =
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav=
a:308)
at =
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni=
toring.java:93)
at =
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
at =
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:=
77)
at =
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142)
at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617)
at java.lang.Thread.run(Thread.java:745)
, Custom Event ID: -1, Message: Host glusterp1 is non responsive.
2016-10-27 13:48:25,019 INFO =
[org.ovirt.engine.core.bll.VdsEventListener] =
(org.ovirt.thread.pool-8-thread-29) [3877c2ef] =
ResourceManager::vdsNotResponding entered for Host =
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'
2016-10-27 13:48:25,054 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
2016-10-27 13:48:25,069 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
VdsNotRespondingTreatmentCommand internal: true. Entities affected : =
ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,079 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
SshSoftFencingCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:25,081 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing =
session on host '192.168.1.31'
2016-10-27 13:48:25,189 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft =
Fencing command on host '192.168.1.31'
2016-10-27 13:48:27,914 INFO =
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp =
Reactor) [] Connecting to /192.168.1.31 <http://192.168.1.31/>
2016-10-27 13:48:29,379 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D=
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', =
vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: =
46d1a470
2016-10-27 13:48:30,383 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: =
46d1a470
2016-10-27 13:48:30,449 INFO =
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] =
(DefaultQuartzScheduler7) [a8ef973] Running command: =
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities =
affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
2016-10-27 13:48:30,464 INFO =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) =
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities =
affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool
2016-10-27 13:48:30,469 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler7) [1c434304] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8
2016-10-27 13:48:37,296 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
2016-10-27 13:48:37,328 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, =
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom =
Event ID: -1, Message: Host glusterp1 is rebooting.
2016-10-27 13:48:37,336 WARN =
[org.ovirt.engine.core.bll.lock.InMemoryLockManager] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release =
exclusive lock which does not exist, lock key: =
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'
2016-10-27 13:48:37,336 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
2016-10-27 13:49:15,787 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster
2016-10-27 13:50:15,790 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler1) [2d374249] No up server in cluster
=3D=3D=3D=3D=3D=3D=3D
=20
=20
=20
On 27 October 2016 at 13:45, Thing <thing.thing(a)gmail.com =
<mailto:thing.thing@gmail.com>> wrote:
While trying to figure out how to deploy storage I put 1 host into =
maintenance mode, trying to re-activate it and its failed.
=20
It seems to be stuck as neither in activated nor maintenance, so how =
would I go
about fixing this?
=20
So what log(s) would this be written to? =20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">This is pointing to network
connectivity =
issues</div><div class=3D""><br
class=3D""></div><div class=3D"">You can =
look at logs under /var/logs/ovirt-*</div><div class=3D""><br =
class=3D""></div><div class=3D"">Check your dns and
make sure the hosts =
can connect to each other. You can restart the host, mark it as =
restarted in the manager and then activate it.</div><div
class=3D""><br =
class=3D""></div><div class=3D"">Worst case, you can
remove it using =
hosted-engine commands</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Tom</div>
<br class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D"">On Oct 26, 2016, at 8:53 PM, Thing <<a =
href=3D"mailto:thing.thing@gmail.com" =
class=3D"">thing.thing(a)gmail.com</a>&gt; wrote:</div><br
=
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D"">from the engine log I see,<br class=3D""><br =
class=3D"">=3D=3D=3D=3D=3D=3D=3D=3D<br class=3D"">2016-10-27
=
13:45:24,595 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78<br =
class=3D"">2016-10-27 13:46:15,780 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler3) [] No up server in cluster<br =
class=3D"">2016-10-27 13:47:12,326 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot monitoring<br class=3D"">2016-10-27
13:47:12,329 =
INFO [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot configurations monitoring<br
class=3D"">2016-10-27 =
13:47:15,783 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br =
class=3D"">2016-10-27 13:48:15,785 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br =
class=3D"">2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: =
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: =
Message timeout which can be caused by communication issues<br =
class=3D"">2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues<br class=3D"">2016-10-27 13:48:24,879 =
INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] FINISH, =
GetGlusterHostUUIDVDSCommand, log id: 51f83f78<br class=3D"">2016-10-27 =
13:48:24,879 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed: EngineException: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues (Failed with error VDS_NETWORK_ERROR and =
code 5022)<br class=3D"">2016-10-27 13:48:24,887 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call =
Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer =
list from server glusterp1 on Cluster Default.<br class=3D"">2016-10-27 =
13:48:24,908 INFO =
[org.ovirt.engine.core.bll.HandleVdsVersionCommand] =
(DefaultQuartzScheduler2) [6083469d] Running command: =
HandleVdsVersionCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 =
13:48:24,959 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, =
vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1',
=
vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, =
error: VDSGenericException: VDSNetworkException: Message timeout which =
can be caused by communication issues<br class=3D"">2016-10-27 =
13:48:25,003 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, =
Call Stack: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues<br
class=3D""> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe=
turnValue(BrokerCommandBase.java:188)<br
class=3D""> =
at =
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce=
edProxyReturnValue(AbstractGlusterBrokerCommand.java:122)<br =
class=3D""> at =
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu=
teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)<br =
class=3D""> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma=
nd(VdsBrokerCommand.java:110)<br class=3D"">
at =
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa=
se.java:73)<br class=3D""> at =
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)<b=
r class=3D""> at =
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana=
ger.java:451)<br class=3D""> at =
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro=
ntendImpl.java:33)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)=
<br class=3D""> at =
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand=
(StorageHandlingCommandBase.java:657)<br
class=3D""> =
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo=
mmand.java:397)<br class=3D""> at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom=
mand.java:149)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1305)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1447)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
75)<br class=3D""> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre=
ssed(TransactionSupport.java:166)<br
class=3D""> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(=
TransactionSupport.java:105)<br class=3D"">
at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)<br =
class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)<=
br class=3D""> at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)<br =
class=3D""> at =
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)<br =
class=3D""> at =
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)<br =
class=3D""> at =
sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)<br class=3D""> at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70)<br class=3D"">
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80)<br class=3D"">
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93)<br class=3D""> at
=
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro=
cessInvocation(ExecutionTimeInterceptor.java:43)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)<br class=3D""> at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73)<br =
class=3D""> at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio=
nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc=
eptor.java:53)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63)<br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)<=
br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept=
or.processInvocation(CurrentInvocationContextInterceptor.java:41)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI=
nvocation(WaitTimeInterceptor.java:43)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc=
essInvocation(ShutDownInterceptorFactory.java:64)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat=
ion(LoggingInterceptor.java:66)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na=
mespaceContextInterceptor.java:50)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b=
r class=3D""> at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636)<br class=3D""> at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61)<br class=3D""> at
=
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b=
r class=3D""> at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)<br=
class=3D""> at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73)<br class=3D""> at =
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc=
tion(Unknown Source)<br class=3D""> at =
sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav=
a:436)<br class=3D""> at =
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri=
seBeanProxyMethodHandler.java:127)<br
class=3D""> at =
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa=
rgetBeanInstance.java:56)<br class=3D""> at
=
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav=
a:100)<br class=3D""> at =
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac=
kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk=
nown Source)<br class=3D""> at =
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav=
a:308)<br class=3D""> at =
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni=
toring.java:93)<br class=3D""> at =
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)<br=
class=3D""> at =
sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:=
77)<br class=3D""> at =
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)<b=
r class=3D""> at =
org.quartz.core.JobRunShell.run(JobRunShell.java:213)<br =
class=3D""> at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br=
class=3D""> at =
java.util.concurrent.FutureTask.run(FutureTask.java:266)<br =
class=3D""> at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142)<br class=3D""> at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617)<br class=3D""> at =
java.lang.Thread.run(Thread.java:745)<br class=3D"">, Custom Event ID: =
-1, Message: Host glusterp1 is non responsive.<br class=3D"">2016-10-27 =
13:48:25,019 INFO [org.ovirt.engine.core.bll.VdsEventListener] =
(org.ovirt.thread.pool-8-thread-29) [3877c2ef] =
ResourceManager::vdsNotResponding entered for Host =
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'<br =
class=3D"">2016-10-27 13:48:25,054 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:25,069
INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
VdsNotRespondingTreatmentCommand internal: true. Entities affected =
: ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br =
class=3D"">2016-10-27 13:48:25,079 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
SshSoftFencingCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 =
13:48:25,081 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing =
session on host '192.168.1.31'<br class=3D"">2016-10-27
13:48:25,189 =
INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft =
Fencing command on host '192.168.1.31'<br class=3D"">2016-10-27 =
13:48:27,914 INFO =
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp =
Reactor) [] Connecting to /<a href=3D"http://192.168.1.31/" =
class=3D"">192.168.1.31</a><br class=3D"">2016-10-27
13:48:29,379 =
INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D=
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', =
vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: =
46d1a470<br class=3D"">2016-10-27 13:48:30,383 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: =
46d1a470<br class=3D"">2016-10-27 13:48:30,449 INFO =
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] =
(DefaultQuartzScheduler7) [a8ef973] Running command: =
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities =
affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br =
class=3D"">2016-10-27 13:48:30,464 INFO =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) =
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities =
affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: =
StoragePool<br class=3D"">2016-10-27 13:48:30,469 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler7) [1c434304] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8<br =
class=3D"">2016-10-27 13:48:37,296 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:37,328
INFO =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, =
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom =
Event ID: -1, Message: Host glusterp1 is rebooting.<br =
class=3D"">2016-10-27 13:48:37,336 WARN =
[org.ovirt.engine.core.bll.lock.InMemoryLockManager] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release =
exclusive lock which does not exist, lock key: =
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'<br
class=3D"">2016-10-27 =
13:48:37,336 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:49:15,787
INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster<br =
class=3D"">2016-10-27 13:50:15,790 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler1) [2d374249] No up server in cluster<br =
class=3D"">=3D=3D=3D=3D=3D=3D=3D<br class=3D""><br
class=3D""><br =
class=3D""></div><div class=3D"gmail_extra"><br
class=3D""><div =
class=3D"gmail_quote">On 27 October 2016 at 13:45, Thing <span
dir=3D"ltr"=
class=3D""><<a href=3D"mailto:thing.thing@gmail.com"
target=3D"_blank"=
class=3D"">thing.thing(a)gmail.com</a>&gt;</span> wrote:<br
=
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D""><div class=3D""><div
class=3D"">While trying to figure out =
how to deploy storage I put 1 host into maintenance mode, trying to =
re-activate it and its failed.<br class=3D""></div><br
class=3D""></div>It=
seems to be stuck as neither in activated nor maintenance, so how would =
I go about fixing this?<br class=3D""><br class=3D"">So
what log(s) =
would this be written to? <br class=3D""></div>
</blockquote></div><br class=3D""></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_96739A99-580C-46CB-B387-9500F44ECBDA--
6:56 p.m.
--Apple-Mail=_0B076BDD-4E22-43D4-85EA-584A51AD8929
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
What does gluster peer status say on each host?
Tom Gamull
Red Hat | Solutions Architect
(404) 494-0280 | @magicalyak
On Oct 26, 2016, at 8:55 PM, Tom Gamull <tgamull(a)redhat.com>
wrote:
=20
This is pointing to network connectivity issues
=20
You can look at logs under /var/logs/ovirt-*
=20
Check your dns and make sure the hosts can connect to each other. You =
can
restart the host, mark it as restarted in the manager and then =
activate it.
=20
Worst case, you can remove it using hosted-engine commands
=20
Tom
=20
> On Oct 26, 2016, at 8:53 PM, Thing <thing.thing(a)gmail.com =
<mailto:thing.thing@gmail.com>> wrote:
>=20
> from the engine log I see,
>=20
> =3D=3D=3D=3D=3D=3D=3D=3D
> 2016-10-27 13:45:24,595 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78
> 2016-10-27 13:46:15,780 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler3) [] No up server in cluster
> 2016-10-27 13:47:12,326 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot monitoring
> 2016-10-27 13:47:12,329 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot configurations monitoring
> 2016-10-27 13:47:15,783 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
> 2016-10-27 13:48:15,785 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster
> 2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: =
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: =
Message timeout which can be caused by communication issues
> 2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues
> 2016-10-27 13:48:24,879 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] FINISH, =
GetGlusterHostUUIDVDSCommand, log id: 51f83f78
> 2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) =
[3877c2ef] Command 'org.ovirt.engine.core.bll.InitVdsOnUpCommand' =
failed: EngineException: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues (Failed with error VDS_NETWORK_ERROR and =
code 5022)
> 2016-10-27 13:48:24,887 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call =
Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer =
list from server glusterp1 on Cluster Default.
> 2016-10-27 13:48:24,908 INFO =
[org.ovirt.engine.core.bll.HandleVdsVersionCommand] =
(DefaultQuartzScheduler2) [6083469d] Running command: =
HandleVdsVersionCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:24,959 INFO =
[org.ovirt.engine.core.vdsbroker.VdsManager] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, =
vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1',
=
vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, =
error: VDSGenericException: VDSNetworkException: Message timeout which =
can be caused by communication issues
> 2016-10-27 13:48:25,003 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, =
Call Stack: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues
> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe=
turnValue(BrokerCommandBase.java:188)
> at =
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce=
edProxyReturnValue(AbstractGlusterBrokerCommand.java:122)
> at =
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu=
teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)
> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma=
nd(VdsBrokerCommand.java:110)
> at =
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa=
se.java:73)
> at =
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> at =
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana=
ger.java:451)
> at =
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro=
ntendImpl.java:33)
> at =
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)=
> at =
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand=
(StorageHandlingCommandBase.java:657)
> at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo=
mmand.java:397)
> at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom=
mand.java:149)
> at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1305)
> at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1447)
> at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
75)
> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre=
ssed(TransactionSupport.java:166)
> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(=
TransactionSupport.java:105)
> at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
> at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
>> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)
> at =
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)
> at =
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)
>> at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
> at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)
> at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70)
> at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80)
> at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93)
> at =
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro=
cessInvocation(ExecutionTimeInterceptor.java:43)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)
> at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73)
> at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
> at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio=
nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc=
eptor.java:53)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63)
> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept=
or.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI=
nvocation(WaitTimeInterceptor.java:43)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc=
essInvocation(ShutDownInterceptorFactory.java:64)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat=
ion(LoggingInterceptor.java:66)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na=
mespaceContextInterceptor.java:50)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636)
> at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
> at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185)
> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)
> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)
> at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73)
> at =
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc=
tion(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
> at =
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav=
a:436)
> at =
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri=
seBeanProxyMethodHandler.java:127)
> at =
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa=
rgetBeanInstance.java:56)
> at =
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav=
a:100)
> at =
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac=
kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk=
nown Source)
> at =
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav=
a:308)
> at =
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni=
toring.java:93)
> at =
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)
>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
> at =
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:=
77)
> at =
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142)
> at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617)
> at java.lang.Thread.run(Thread.java:745)
> , Custom Event ID: -1, Message: Host glusterp1 is non responsive.
> 2016-10-27 13:48:25,019 INFO =
[org.ovirt.engine.core.bll.VdsEventListener] =
(org.ovirt.thread.pool-8-thread-29) [3877c2ef] =
ResourceManager::vdsNotResponding entered for Host =
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'
> 2016-10-27 13:48:25,054 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
> 2016-10-27 13:48:25,069 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
VdsNotRespondingTreatmentCommand internal: true. Entities affected : =
ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:25,079 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
SshSoftFencingCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:25,081 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing =
session on host '192.168.1.31'
> 2016-10-27 13:48:25,189 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft =
Fencing command on host '192.168.1.31'
> 2016-10-27 13:48:27,914 INFO =
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp =
Reactor) [] Connecting to /192.168.1.31 <http://192.168.1.31/>
> 2016-10-27 13:48:29,379 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D=
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', =
vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: =
46d1a470
> 2016-10-27 13:48:30,383 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: =
46d1a470
> 2016-10-27 13:48:30,449 INFO =
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] =
(DefaultQuartzScheduler7) [a8ef973] Running command: =
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities =
affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS
> 2016-10-27 13:48:30,464 INFO =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) =
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities =
affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: StoragePool
> 2016-10-27 13:48:30,469 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler7) [1c434304] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8
> 2016-10-27 13:48:37,296 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
> 2016-10-27 13:48:37,328 INFO =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, =
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom =
Event ID: -1, Message: Host glusterp1 is rebooting.
> 2016-10-27 13:48:37,336 WARN =
[org.ovirt.engine.core.bll.lock.InMemoryLockManager] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release =
exclusive lock which does not exist, lock key: =
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'
> 2016-10-27 13:48:37,336 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<VD=
S_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'
> 2016-10-27 13:49:15,787 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster
> 2016-10-27 13:50:15,790 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler1) [2d374249] No up server in cluster
> =3D=3D=3D=3D=3D=3D=3D
>=20
>=20
>=20
> On 27 October 2016 at 13:45, Thing <thing.thing(a)gmail.com =
<mailto:thing.thing@gmail.com>> wrote:
> While trying to figure out how to deploy storage I put 1 host
into =
maintenance mode, trying to re-activate it and its failed.
>=20
> It seems to be stuck as neither in activated nor maintenance, so how =
would I
go about fixing this?
>=20
> So what log(s) would this be written to? =20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
=20
--Apple-Mail=_0B076BDD-4E22-43D4-85EA-584A51AD8929
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">What does gluster peer status say on each host?<br =
class=3D""><div class=3D"">
<div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: =
normal; letter-spacing: normal; orphans: auto; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; widows: =
auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"
class=3D""><span=
style=3D"color: rgb(34, 34, 34); font-family: arial, sans-serif; =
font-size: 13px; background-color: rgb(255, 255, 255);"
class=3D""><br =
class=3D"Apple-interchange-newline">T</span><span
style=3D"color: =
rgb(34, 34, 34); font-family: arial, sans-serif; font-size: 13px; =
background-color: rgb(255, 255, 255);" class=3D"">om =
Gamull</span></div><div style=3D"color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; orphans: auto; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: =
0px;" class=3D""><div style=3D"color: rgb(34, 34, 34);
font-family: =
arial, sans-serif; font-size: 13px;" class=3D"">Red Hat | Solutions =
Architect</div><div style=3D"color: rgb(34, 34, 34); font-family: arial, =
sans-serif; font-size: 13px;" class=3D"">(404) 494-0280 | =
@magicalyak</div></div>
</div>
<br class=3D""><div style=3D""><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On Oct 26, 2016, at 8:55 PM, Tom Gamull <<a =
href=3D"mailto:tgamull@redhat.com"
class=3D"">tgamull(a)redhat.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D""><meta =
http-equiv=3D"Content-Type" content=3D"text/html charset=3Dus-ascii"
=
class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: =
space; -webkit-line-break: after-white-space;" class=3D""><div =
class=3D"">This is pointing to network connectivity issues</div><div
=
class=3D""><br class=3D""></div><div
class=3D"">You can look at logs =
under /var/logs/ovirt-*</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Check your dns and make sure the hosts can connect to each =
other. You can restart the host, mark it as restarted in the =
manager and then activate it.</div><div class=3D""><br =
class=3D""></div><div class=3D"">Worst case, you can
remove it using =
hosted-engine commands</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Tom</div>
<br class=3D""><div class=3D""><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On Oct 26, 2016, at 8:53 PM, Thing <<a =
href=3D"mailto:thing.thing@gmail.com" =
class=3D"">thing.thing(a)gmail.com</a>&gt; wrote:</div><br
=
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D"">from the engine log I see,<br class=3D""><br =
class=3D"">=3D=3D=3D=3D=3D=3D=3D=3D<br class=3D"">2016-10-27
=
13:45:24,595 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 51f83f78<br =
class=3D"">2016-10-27 13:46:15,780 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler3) [] No up server in cluster<br =
class=3D"">2016-10-27 13:47:12,326 INFO =
[org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot monitoring<br class=3D"">2016-10-27
13:47:12,329 =
INFO [org.ovirt.engine.core.bll.gluster.GlusterSnapshotSyncJob] =
(DefaultQuartzScheduler10) [76412dcc] No UP server found in cluster =
'Default' for snapshot configurations monitoring<br
class=3D"">2016-10-27 =
13:47:15,783 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br =
class=3D"">2016-10-27 13:48:15,785 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler4) [27554ab4] No up server in cluster<br =
class=3D"">2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: null, Call Stack: =
null, Custom Event ID: -1, Message: VDSM glusterp1 command failed: =
Message timeout which can be caused by communication issues<br =
class=3D"">2016-10-27 13:48:24,879 ERROR =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'})' execution failed: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues<br class=3D"">2016-10-27 13:48:24,879 =
INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler2) [3877c2ef] FINISH, =
GetGlusterHostUUIDVDSCommand, log id: 51f83f78<br class=3D"">2016-10-27 =
13:48:24,879 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] =
(DefaultQuartzScheduler2) [3877c2ef] Command =
'org.ovirt.engine.core.bll.InitVdsOnUpCommand' failed: EngineException: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues (Failed with error VDS_NETWORK_ERROR and =
code 5022)<br class=3D"">2016-10-27 13:48:24,887 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler2) [3877c2ef] Correlation ID: 3877c2ef, Call =
Stack: null, Custom Event ID: -1, Message: Failed to fetch gluster peer =
list from server glusterp1 on Cluster Default.<br class=3D"">2016-10-27 =
13:48:24,908 INFO =
[org.ovirt.engine.core.bll.HandleVdsVersionCommand] =
(DefaultQuartzScheduler2) [6083469d] Running command: =
HandleVdsVersionCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 =
13:48:24,959 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Server failed to respond, =
vds_id=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', vds_name=3D'glusterp1',
=
vm_count=3D0, spm_status=3D'None', non-responsive_timeout (seconds)=3D60, =
error: VDSGenericException: VDSNetworkException: Message timeout which =
can be caused by communication issues<br class=3D"">2016-10-27 =
13:48:25,003 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-42) [3877c2ef] Correlation ID: null, =
Call Stack: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Message timeout which can be =
caused by communication issues<br
class=3D""> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyRe=
turnValue(BrokerCommandBase.java:188)<br
class=3D""> =
at =
org.ovirt.engine.core.vdsbroker.gluster.AbstractGlusterBrokerCommand.proce=
edProxyReturnValue(AbstractGlusterBrokerCommand.java:122)<br =
class=3D""> at =
org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand.execu=
teVdsBrokerCommand(GetGlusterHostUUIDVDSCommand.java:19)<br =
class=3D""> at =
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSComma=
nd(VdsBrokerCommand.java:110)<br class=3D"">
at =
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBa=
se.java:73)<br class=3D""> at =
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)<b=
r class=3D""> at =
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceMana=
ger.java:451)<br class=3D""> at =
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFro=
ntendImpl.java:33)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171)=
<br class=3D""> at =
org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand=
(StorageHandlingCommandBase.java:657)<br
class=3D""> =
at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.initGlusterHost(InitVdsOnUpCo=
mmand.java:397)<br class=3D""> at =
org.ovirt.engine.core.bll.InitVdsOnUpCommand.executeCommand(InitVdsOnUpCom=
mand.java:149)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1305)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1447)<br class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
75)<br class=3D""> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre=
ssed(TransactionSupport.java:166)<br
class=3D""> at =
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(=
TransactionSupport.java:105)<br class=3D"">
at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)<br =
class=3D""> at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)<=
br class=3D""> at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493)<br =
class=3D""> at =
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475)<br =
class=3D""> at =
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:419)<br =
class=3D""> at =
sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)<br class=3D""> at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70)<br class=3D"">
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80)<br class=3D"">
at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93)<br class=3D""> at
=
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro=
cessInvocation(ExecutionTimeInterceptor.java:43)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437)<br class=3D""> at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73)<br =
class=3D""> at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio=
nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc=
eptor.java:53)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63)<br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)<=
br class=3D""> at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept=
or.processInvocation(CurrentInvocationContextInterceptor.java:41)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI=
nvocation(WaitTimeInterceptor.java:43)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc=
essInvocation(ShutDownInterceptorFactory.java:64)<br =
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat=
ion(LoggingInterceptor.java:66)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na=
mespaceContextInterceptor.java:50)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64)<br
class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b=
r class=3D""> at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636)<br class=3D""> at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61)<br class=3D""> at
=
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)<b=
r class=3D""> at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80)<br
class=3D""> =
at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)<br=
class=3D""> at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185)<br class=3D""> at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0)<br class=3D""> at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61)<br class=3D""> at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73)<br class=3D""> at =
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAc=
tion(Unknown Source)<br class=3D""> at =
sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.jav=
a:436)<br class=3D""> at =
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(Enterpri=
seBeanProxyMethodHandler.java:127)<br
class=3D""> at =
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTa=
rgetBeanInstance.java:56)<br class=3D""> at
=
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.jav=
a:100)<br class=3D""> at =
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$Bac=
kendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unk=
nown Source)<br class=3D""> at =
org.ovirt.engine.core.bll.VdsEventListener.vdsUpEvent(VdsEventListener.jav=
a:308)<br class=3D""> at =
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMoni=
toring.java:93)<br class=3D""> at =
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238)<br=
class=3D""> at =
sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)<br =
class=3D""> at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)<br class=3D""> at =
java.lang.reflect.Method.invoke(Method.java:498)<br =
class=3D""> at =
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:=
77)<br class=3D""> at =
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)<b=
r class=3D""> at =
org.quartz.core.JobRunShell.run(JobRunShell.java:213)<br =
class=3D""> at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br=
class=3D""> at =
java.util.concurrent.FutureTask.run(FutureTask.java:266)<br =
class=3D""> at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142)<br class=3D""> at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617)<br class=3D""> at =
java.lang.Thread.run(Thread.java:745)<br class=3D"">, Custom Event ID: =
-1, Message: Host glusterp1 is non responsive.<br class=3D"">2016-10-27 =
13:48:25,019 INFO [org.ovirt.engine.core.bll.VdsEventListener] =
(org.ovirt.thread.pool-8-thread-29) [3877c2ef] =
ResourceManager::vdsNotResponding entered for Host =
'260c0a92-2856-4cd6-a784-01ac95fc41d5', '192.168.1.31'<br =
class=3D"">2016-10-27 13:48:25,054 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock Acquired to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:25,069
INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
VdsNotRespondingTreatmentCommand internal: true. Entities affected =
: ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br =
class=3D"">2016-10-27 13:48:25,079 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Running command: =
SshSoftFencingCommand internal: true. Entities affected : ID: =
260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br class=3D"">2016-10-27 =
13:48:25,081 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Opening SSH Soft Fencing =
session on host '192.168.1.31'<br class=3D"">2016-10-27
13:48:25,189 =
INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Executing SSH Soft =
Fencing command on host '192.168.1.31'<br class=3D"">2016-10-27 =
13:48:27,914 INFO =
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp =
Reactor) [] Connecting to /<a href=3D"http://192.168.1.31/" =
class=3D"">192.168.1.31</a><br class=3D"">2016-10-27
13:48:29,379 =
INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] START, GetHardwareInfoVDSCommand(HostName =3D=
glusterp1, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5', =
vds=3D'Host[glusterp1,260c0a92-2856-4cd6-a784-01ac95fc41d5]'}), log id: =
46d1a470<br class=3D"">2016-10-27 13:48:30,383 INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] =
(DefaultQuartzScheduler7) [] FINISH, GetHardwareInfoVDSCommand, log id: =
46d1a470<br class=3D"">2016-10-27 13:48:30,449 INFO =
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] =
(DefaultQuartzScheduler7) [a8ef973] Running command: =
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities =
affected : ID: 260c0a92-2856-4cd6-a784-01ac95fc41d5 Type: VDS<br =
class=3D"">2016-10-27 13:48:30,464 INFO =
[org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) =
[1c434304] Running command: InitVdsOnUpCommand internal: true. Entities =
affected : ID: 58098515-0126-0151-02eb-0000000003cb Type: =
StoragePool<br class=3D"">2016-10-27 13:48:30,469 INFO =
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterHostUUIDVDSCommand] =
(DefaultQuartzScheduler7) [1c434304] START, =
GetGlusterHostUUIDVDSCommand(HostName =3D glusterp1, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'260c0a92-2856-4cd6-a784-01ac95fc41d5'}), log id: 4ca089d8<br =
class=3D"">2016-10-27 13:48:37,296 INFO =
[org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:48:37,328
INFO =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Correlation ID: 43df6659, =
Job ID: 091c3b5f-d8e6-4200-b8fb-39d0d8b5b330, Call Stack: null, Custom =
Event ID: -1, Message: Host glusterp1 is rebooting.<br =
class=3D"">2016-10-27 13:48:37,336 WARN =
[org.ovirt.engine.core.bll.lock.InMemoryLockManager] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Trying to release =
exclusive lock which does not exist, lock key: =
'260c0a92-2856-4cd6-a784-01ac95fc41d5VDS_FENCE'<br
class=3D"">2016-10-27 =
13:48:37,336 INFO =
[org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] =
(org.ovirt.thread.pool-8-thread-29) [43df6659] Lock freed to object =
'EngineLock:{exclusiveLocks=3D'[260c0a92-2856-4cd6-a784-01ac95fc41d5=3D<=
;VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', =
sharedLocks=3D'null'}'<br class=3D"">2016-10-27 13:49:15,787
INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler8) [beed7f3] No up server in cluster<br =
class=3D"">2016-10-27 13:50:15,790 INFO =
[org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService] =
(DefaultQuartzScheduler1) [2d374249] No up server in cluster<br =
class=3D"">=3D=3D=3D=3D=3D=3D=3D<br class=3D""><br
class=3D""><br =
class=3D""></div><div class=3D"gmail_extra"><br
class=3D""><div =
class=3D"gmail_quote">On 27 October 2016 at 13:45, Thing <span
dir=3D"ltr"=
class=3D""><<a href=3D"mailto:thing.thing@gmail.com"
target=3D"_blank"=
class=3D"">thing.thing(a)gmail.com</a>&gt;</span> wrote:<br
=
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D""><div class=3D""><div
class=3D"">While trying to figure out =
how to deploy storage I put 1 host into maintenance mode, trying to =
re-activate it and its failed.<br class=3D""></div><br
class=3D""></div>It=
seems to be stuck as neither in activated nor maintenance, so how would =
I go about fixing this?<br class=3D""><br class=3D"">So
what log(s) =
would this be written to? <br class=3D""></div>
</blockquote></div><br class=3D""></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... =
class=3D""></div></blockquote></div><br =
class=3D""></div></div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_0B076BDD-4E22-43D4-85EA-584A51AD8929--
2748
days inactive
2748
days old
3 comments
2 participants
participants (2)
-
Thing -
Tom Gamull