Hi I'm looking for some help with an issue with one of our hypervisors We recently upgraded one of our hypervisors from ovirt 4.0 to 4.1 and Centos 7.3.1611 to 7.4.1708 using the following procedure: 1. Migrated all VMs off hypervisor 2. Put hypervisor into maintenance mode and deleted it from the cluster 3. Re-subscribed the hypervisor in katello to the new OS channels 4. Upgraded the OS to 7.4.1708 5. After reboot subscribed the hypervisor to the new ovirt 4.1 channels 6. Added the hypervisor back into the cluster (this installed the new ovirt packages) 7. Returned hypervisor to maintenance mode 8. Applied any remaining updates and rebooted 9. Checked all required networks were assigned to the hypervisor 10. Took the hypervisor out of maintenance mode After 5 mins of the hypervisor sitting in 'Activating' it finally got into an up state, however we get an error in the event log: Failed to activate Host <node>.(User: <user>@<domin>) Looking in the engine log I can see the following: 2018-06-27 12:32:43,445 ERROR [org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery] (org.ovirt.thread.pool-6-thread-48) [] Query 'GetVdsInterfacesByVdsIdQuery' failed: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction 2018-06-27 12:32:43,445 ERROR [org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery] (org.ovirt.thread.pool-6-thread-48) [] Exception: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762) [spring-jdbc.jar:4.2.4.RELEASE] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.jar:4.2.4.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:109) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:101) [dal.jar:] at org.ovirt.engine.core.dao.ClusterDaoImpl.get(ClusterDaoImpl.java:52) [dal.jar:] at org.ovirt.engine.core.dao.ClusterDaoImpl.get(ClusterDaoImpl.java:44) [dal.jar:] at org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.build(NetworkImplementationDetailsUtils.java:72) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.build(NetworkImplementationDetailsUtils.java:85) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.NetworkImplementationDetailsUtils.calculateNetworkImplementationDetails(NetworkImplementationDetailsUtils.java:58) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.GetVdsInterfacesByVdsIdQuery.executeQueryCommand(GetVdsInterfacesByVdsIdQuery.java:86) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:103) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:558) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalQuery(Backend.java:524) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalQuery(Backend.java:734) [bll.jar:] at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source) [:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) [:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:376) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view4.runInternalQuery(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor946.invoke(Unknown Source) [:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:128) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalQuery(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.getHostInterfaces(NetworkClusterHelper.java:147) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.atLeastOneHostDoesNotHaveNetworkAttached(NetworkClusterHelper.java:131) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.updateStatusOfRequiredNetworkCluster(NetworkClusterHelper.java:115) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.setStatus(NetworkClusterHelper.java:97) [bll.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand.lambda$executeCommand$0(ActivateVdsCommand.java:71) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [utils.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:67) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141] Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146) at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:4.2.4.RELEASE] ... 119 more Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:424) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747) at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ... 122 more Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffffac12138e:19e6ae22:5b3234b8:4595d status: ActionStatus.ABORTED > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:408) ... 124 more 2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (org.ovirt.thread.pool-6-thread-48) [] transaction rolled back 2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Activate host finished. Lock released. Monitoring can run now for host '<host>' from data-center '<DC>' 2018-06-27 12:32:43,449 ERROR [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.bll.ActivateVdsCommand' failed: null 2018-06-27 12:32:43,449 ERROR [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Exception: java.lang.NullPointerException at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.atLeastOneHostDoesNotHaveNetworkAttached(NetworkClusterHelper.java:133) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.updateStatusOfRequiredNetworkCluster(NetworkClusterHelper.java:115) [bll.jar:] at org.ovirt.engine.core.bll.network.cluster.NetworkClusterHelper.setStatus(NetworkClusterHelper.java:97) [bll.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand.lambda$executeCommand$0(ActivateVdsCommand.java:71) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [utils.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:67) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:] at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141] Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146) at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:4.2.4.RELEASE] ... 119 more Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:424) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747) at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ... 122 more Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffffac12138e:19e6ae22:5b3234b8:4595d status: ActionStatus.ABORTED > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:408) ... 124 more 2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (org.ovirt.thread.pool-6-thread-48) [] transaction rolled back 2018-06-27 12:32:43,449 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Activate host finished. Lock released. Monitoring can run now for host '<host>' from data-center '<DC>'ine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.bll.ActivateVdsCommand' failed: null 4186-bf54-17123ca3c491, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Host <host>.(User: <user>@<domain>). 2018-06-27 12:32:43,464 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Lock freed to object 'EngineLock:{exclusiveLocks='[d35d9e0c-3c83-4a9d-8f8e-b72df5ea521c=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2018-06-27 12:32:47,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [e83484a] START, GetHardwareInfoVDSCommand(HostName = <host>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='<ID>', vds='Host[<host>]'}), log id: 3b9e6de5 2018-06-27 12:32:47,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [e83484a] FINISH, GetHardwareInfoVDSCommand, log id: 3b9e6de5 2018-06-27 12:32:47,837 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [233b0ff4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: <ID> Type: VDS 2018-06-27 12:32:47,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2d5a709e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: <ID> Type: StoragePool 2018-06-27 12:32:47,863 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [257c0825] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: <ID> Type: StoragePool 2018-06-27 12:32:47,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [257c0825] START, ConnectStorageServerVDSCommand(HostName = <host>, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='<ID>', storagePoolId='<ID>', storageType='FCP', connectionList='[StorageServerConnections:{id='00000000-0000-0000-0000-000000000000', connection='null', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 797eeb05 at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_141] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141] 2018-06-27 12:32:43,458 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] Correlation ID: 4fd81e15, Job ID: d2504aa4-cded-4186-bf54-17123ca3c491, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Host <host>.(User: <user>@<domian>). 2018-06-27 12:32:43,464 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-48) [] Lock freed to object 'EngineLock:{exclusiveLocks='[<ID>=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2018-06-27 12:32:47,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [e83484a] START, GetHardwareInfoVDSCommand(HostName = <host>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='<ID>', vds='Host[<host>]'}), log id: 3b9e6de5 2018-06-27 12:32:47,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [e83484a] FINISH, GetHardwareInfoVDSCommand, log id: 3b9e6de5 2018-06-27 12:32:47,837 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [233b0ff4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: <ID> Type: VDS 2018-06-27 12:32:47,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2d5a709e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 57fe604a-010a-01af-0280-0000000001d0 Type: StoragePool At the same time I see these in the engine server.log: 2018-06-27 12:32:43,436 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffac12138e:19e6ae22:5b3234b8:4595d in state RUN 2018-06-27 12:32:43,438 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffffac12138e:19e6ae22:5b3234b8:4595d invoked while multiple threads active within it. 2018-06-27 12:32:43,439 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffffac12138e:19e6ae22:5b3234b8:4595d completed with multiple threads - thread org.ovirt.thread.pool-6-thread-48 was in progress with java.math.BigInteger.squareToLen(BigInteger.java:1915) 2018-06-27 12:32:43,439 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffffac12138e:19e6ae22:5b3234b8:4595d aborting with 1 threads active! 2018-06-27 12:32:43,441 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffffac12138e:19e6ae22:5b3234b8:4595d 2018-06-27 12:32:43,449 WARN [com.arjuna.ats.arjuna] (org.ovirt.thread.pool-6-thread-48) ARJUNA012077: Abort called on already aborted atomic action 0:ffffac12138e:19e6ae22:5b3234b8:4595d Any help is resolving this issue would be appricated Thanks