1:24 p.m.
Hi I'm looking for some help with an issue with one of our hypervisors
We recently upgraded one of our hypervisors from ovirt 4.0 to 4.1 and Centos 7.3.1611 to
7.4.1708 using the following procedure:
1. Migrated all VMs off hypervisor
2. Put hypervisor into maintenance mode and deleted it from the cluster
3. Re-subscribed the hypervisor in katello to the new OS channels
4. Upgraded the OS to 7.4.1708
5. After reboot subscribed the hypervisor to the new ovirt 4.1 channels
6. Added the hypervisor back into the cluster (this installed the new ovirt packages)
7. Returned hypervisor to maintenance mode
8. Applied any remaining updates and rebooted
9. Checked all required networks were assigned to the hypervisor
10. Took the hypervisor out of maintenance mode
After 5 mins of the hypervisor sitting in 'Activating' it finally got into an up
state, however we get an error in the event log:
Failed to activate Host <node>.(User: <user>@<domin>)
Looking in the engine log I can see the following:
2018-06-27 12:32:43,445 ERROR
[org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery]
(org.ovirt.thread.pool-6-thread-48) [] Query 'GetVdsInterfacesByVdsIdQuery'
failed: Could not get JDBC Connection; nested exception is java.sql.SQLException:
javax.resource.ResourceException: IJ000460: Error checking for a transaction
2018-06-27 12:32:43,445 ERROR
[org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery]
(org.ovirt.thread.pool-6-thread-48) [] Exception:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection;
nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460:
Error checking for a transaction
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
[spring-jdbc.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
[spring-jdbc.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680)
[spring-jdbc.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712)
[spring-jdbc.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762)
[spring-jdbc.jar:4.2.4.RELEASE]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154)
[dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120)
[dal.jar:]
at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
[spring-jdbc.jar:4.2.4.RELEASE]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147)
[dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:109)
[dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:101)
[dal.jar:]
at org.ovirt.engine.core.dao.ClusterDaoImpl.get(ClusterDaoImpl.java:52)
[dal.jar:]
at org.ovirt.engine.core.dao.ClusterDaoImpl.get(ClusterDaoImpl.java:44)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.build(NetworkImplementationDetailsUtils.java:72)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.build(NetworkImplementationDetailsUtils.java:85)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.calculateNetworkImplementationDetails(NetworkImplementationDetailsUtils.java:58)
[vdsbroker.jar:]
at
org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery.executeQueryCommand(GetVdsInterfacesByVdsIdQuery.java:86)
[bll.jar:]
at
org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:103)
[bll.jar:]
at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:558) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalQuery(Backend.java:524)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalQuery(Backend.java:734)
[bll.jar:]
at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source) [:1.8.0_141]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
[bll.jar:]
at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) [:1.8.0_141]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:376)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view4.runInternalQuery(Unknown
Source) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor946.invoke(Unknown Source) [:1.8.0_141]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:128)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalQuery(Unknown
Source) [bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.getHostInterfaces(NetworkClusterHelper.java:147)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.atLeastOneHostDoesNotHaveNetworkAttached(NetworkClusterHelper.java:131)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.updateStatusOfRequiredNetworkCluster(NetworkClusterHelper.java:115)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.setStatus(NetworkClusterHelper.java:97)
[bll.jar:]
at
org.ovirt.engine.core.bll.ActivateVdsCommand.lambda$executeCommand$0(ActivateVdsCommand.java:71)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204)
[utils.jar:]
at
org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:67)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
[utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
[utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92)
[utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error
checking for a transaction
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
at
org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
[spring-jdbc.jar:4.2.4.RELEASE]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
[spring-jdbc.jar:4.2.4.RELEASE]
... 119 more
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:424)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
... 122 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active:
tx=TransactionImple < ac, BasicAction: 0:ffffac12138e:19e6ae22:5b3234b8:4595d status:
ActionStatus.ABORTED >
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:408)
... 124 more
2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport]
(org.ovirt.thread.pool-6-thread-48) [] transaction rolled back
2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Activate host finished. Lock released. Monitoring
can run now for host '<host>' from data-center '<DC>'
2018-06-27 12:32:43,449 ERROR [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Command
'org.ovirt.engine.core.bll.ActivateVdsCommand' failed: null
2018-06-27 12:32:43,449 ERROR [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Exception: java.lang.NullPointerException
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.atLeastOneHostDoesNotHaveNetworkAttached(NetworkClusterHelper.java:133)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.updateStatusOfRequiredNetworkCluster(NetworkClusterHelper.java:115)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.setStatus(NetworkClusterHelper.java:97)
[bll.jar:]
at
org.ovirt.engine.core.bll.ActivateVdsCommand.lambda$executeCommand$0(ActivateVdsCommand.java:71)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204)
[utils.jar:]
at
org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:67)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
[utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
[utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176)
[bll.jar:]
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92)
[utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error
checking for a transaction
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
at
org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
[spring-jdbc.jar:4.2.4.RELEASE]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
[spring-jdbc.jar:4.2.4.RELEASE]
... 119 more
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:424)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
... 122 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active:
tx=TransactionImple < ac, BasicAction: 0:ffffac12138e:19e6ae22:5b3234b8:4595d status:
ActionStatus.ABORTED > at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:408)
... 124 more
2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport]
(org.ovirt.thread.pool-6-thread-48) [] transaction rolled back
2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Activate host finished. Lock released. Monitoring
can run now for host '<host>' from data-center
'<DC>'ine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48)
[] Command 'org.ovirt.engine.core.bll.ActivateVdsCommand' failed: null
4186-bf54-17123ca3c491, Call Stack: null, Custom Event ID: -1, Message: Failed to activate
Host <host>.(User: <user>@<domain>).
2018-06-27 12:32:43,464 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Lock freed to object
'EngineLock:{exclusiveLocks='[d35d9e0c-3c83-4a9d-8f8e-b72df5ea521c=<VDS,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2018-06-27 12:32:47,033 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler2) [e83484a] START, GetHardwareInfoVDSCommand(HostName =
<host>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='<ID>', vds='Host[<host>]'}), log id: 3b9e6de5
2018-06-27 12:32:47,035 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler2) [e83484a] FINISH, GetHardwareInfoVDSCommand, log id: 3b9e6de5
2018-06-27 12:32:47,837 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(DefaultQuartzScheduler2) [233b0ff4] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
<ID> Type: VDS
2018-06-27 12:32:47,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(DefaultQuartzScheduler2) [2d5a709e] Running command: InitVdsOnUpCommand internal: true.
Entities affected : ID: <ID> Type: StoragePool
2018-06-27 12:32:47,863 INFO
[org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand]
(DefaultQuartzScheduler2) [257c0825] Running command:
ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: <ID>
Type: StoragePool
2018-06-27 12:32:47,879 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(DefaultQuartzScheduler2) [257c0825] START, ConnectStorageServerVDSCommand(HostName =
<host>, StorageServerConnectionManagementVDSParameters:{runAsync='true',
hostId='<ID>', storagePoolId='<ID>',
storageType='FCP',
connectionList='[StorageServerConnections:{id='00000000-0000-0000-0000-000000000000',
connection='null', iqn='null', vfsType='null',
mountOptions='null', nfsVersion='null', nfsRetrans='null',
nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log
id: 797eeb05
at
org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92)
[utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
2018-06-27 12:32:43,458 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-48) [] Correlation ID: 4fd81e15, Job ID:
d2504aa4-cded-4186-bf54-17123ca3c491, Call Stack: null, Custom Event ID: -1, Message:
Failed to activate Host <host>.(User: <user>@<domian>).
2018-06-27 12:32:43,464 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-6-thread-48) [] Lock freed to object
'EngineLock:{exclusiveLocks='[<ID>=<VDS,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2018-06-27 12:32:47,033 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler2) [e83484a] START, GetHardwareInfoVDSCommand(HostName =
<host>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='<ID>', vds='Host[<host>]'}), log id: 3b9e6de5
2018-06-27 12:32:47,035 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler2) [e83484a] FINISH, GetHardwareInfoVDSCommand, log id: 3b9e6de5
2018-06-27 12:32:47,837 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(DefaultQuartzScheduler2) [233b0ff4] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
<ID> Type: VDS
2018-06-27 12:32:47,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(DefaultQuartzScheduler2) [2d5a709e] Running command: InitVdsOnUpCommand internal: true.
Entities affected : ID: 57fe604a-010a-01af-0280-0000000001d0 Type: StoragePool
At the same time I see these in the engine server.log:
2018-06-27 12:32:43,436 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117:
TransactionReaper::check timeout for TX 0:ffffac12138e:19e6ae22:5b3234b8:4595d in state
RUN
2018-06-27 12:32:43,438 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0)
ARJUNA012095: Abort of action id 0:ffffac12138e:19e6ae22:5b3234b8:4595d invoked while
multiple threads active within it.
2018-06-27 12:32:43,439 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0)
ARJUNA012381: Action id 0:ffffac12138e:19e6ae22:5b3234b8:4595d completed with multiple
threads - thread org.ovirt.thread.pool-6-thread-48 was in progress with
java.math.BigInteger.squareToLen(BigInteger.java:1915)
2018-06-27 12:32:43,439 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0)
ARJUNA012108: CheckedAction::check - atomic action 0:ffffac12138e:19e6ae22:5b3234b8:4595d
aborting with 1 threads active!
2018-06-27 12:32:43,441 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0)
ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker
0,5,main] successfully canceled TX 0:ffffac12138e:19e6ae22:5b3234b8:4595d
2018-06-27 12:32:43,449 WARN [com.arjuna.ats.arjuna] (org.ovirt.thread.pool-6-thread-48)
ARJUNA012077: Abort called on already aborted atomic action
0:ffffac12138e:19e6ae22:5b3234b8:4595d
Any help is resolving this issue would be appricated
Thanks