[ovirt-users] Re: Host install failed - cannot set maintenance or remove

I've attached a couple of screenshots, the hosts seem to not succeed in going into non-operational mode. Required network is now unticked. Storage is definitely accessible by all nodes + engine. The moment they succeed into going non-operational, they cycle back to activating. It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` = 'virtA003.cluster'; 2018-05-08 14:54:32,351+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 313cfe33 2018-05-08 14:54:32,353+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory -engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[ 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}' 2018-05-08 14:54:32,385+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact ory-engineScheduled-Thread-30) [32499b7b] Running command: SetNonOperationalVdsCommand internal: true. Entitie s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS 2018-05-08 14:54:32,388+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName = virtA002.cluster, SetVdsSt atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7', status='NonOperational', nonOperation alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b5138fd 2018-05-08 14:54:32,705+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac tory-engineScheduled-Thread-36) [3c65dca0] FINISH, SetVdsStatusVDSCommand, log id: 44441e64 2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set to Non-Operational, it is missing the following networks: 'ovirtmgmt' 2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-Ma nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host v irtA003.cluster does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' 2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler] (EE-ManagedThreadFactory-eng ineScheduled-Thread-36) [3c65dca0] Exception: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ00045 7: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxCo nnectionListener@769d34fc[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnec tion@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=15257 87372702 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=Semaphor eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@3aa2 9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQ L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring- jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619) [spring-jdbc.jar:4.3.9.RE LEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684) [spring-jdbc.jar:4.3.9.RELE ASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716) [spring-jdbc.jar:4.3.9.RELE ASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766) [spring-jdbc.jar:4.3.9.RELE ASE] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallIntern al(PostgresDbEngineDialect.java:152) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(Postgre sDbEngineDialect.java:118) [dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.j ar:4.3.9.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:1 35) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.ja va:105) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:9 7) [dal.jar:] at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149) [dal.jar:] at org.ovirt.engine.core.bll.job.ExecutionHandler.checkIfJobHasTasks(ExecutionHandler.java:893) [bll.j ar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto r.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0 _161] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterce ptor.java:78) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterce ptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsIntercepto r.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFa ctory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionT imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextIntercep tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDi spatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvoc ation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255) [wildfly-ejb3-11. 0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381) [wildfly-ejb3-11.0.0.Fina l.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11 .0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScop eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivat ionInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(Curr entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterce ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor. java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInter ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInt erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.ja va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor .java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor. java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll .jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0 _161] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2. 4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler. java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPoi ntPropagatingEnterpriseTargetBeanInstance.java:67) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2. 4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Prox y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303) [bll.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.setNonOperational(HostNe tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.enforceNetworkCompliance (HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99) [vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSuppo rt.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.jav a:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:1 05) [utils.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.processRefreshCapabilitiesResponse(VdsManager.java:794) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$BeforeFirstRefreshTreatmentCallback.onRes ponse(HostMonitoring.java:767) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45) [vdsbroker.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$processResponse$1(JsonRpcClient.java:182) [vdsm- jsonrpc-java-client.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_161] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161] at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141) [jav ax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut ureTask.run(ManagedScheduledThreadPoolExecutor.java:532) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactory Impl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronMana gedThreadFactory.java:78) Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedCo nnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NOR MAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.j boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueueManagedConnec tionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co nnectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=jav a:/ENGINEDataSource] txSync=null] at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146) at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.jav a:64) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spri ng-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring- jdbc.jar:4.3.9.RELEASE] ... 107 more Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() c l=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NORMAL managed connection= org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0 lastReturned=1525787672 740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.jboss.jca.core.connectio nmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=E NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc connectionManager=5d6abc6 c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] tx Sync=null] at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractC onnectionManager.java:975) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectio nManager.java:792) at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ... 110 more Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:561) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:970) ... 112 more Caused by: java.lang.IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:296) at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:554) ... 113 more 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] transaction rolled back 2018-05-08 14:54:32,749+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock acquired, from now a monitoring of host will be skipped for host 'virtA003.cluster' from data-center 'Default' 2018-05-08 14:54:32,768+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 635dd4c4 2018-05-08 14:54:32,771+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH, SetVdsStatusVDSCommand, log id: 635dd4c4 2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate host finished. Lock released. Monitoring can run now for host 'virtA003.cluster' from data-center 'Default' 2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed to object 'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]', sharedLocks=''}' 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START, GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster, VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log id: 710291f0 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 710291f0 2018-05-08 14:54:35,774+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS 2018-05-08 14:54:35,776+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 707d7be2 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1 hosts 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering hosts id: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster 2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]', sharedLocks=''}' 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running command: ActivateVdsCommand internal: true. Entities affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before acquiring lock in order to prevent monitoring for host 'virtA002.cluster' from data-center 'Default' 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}' 2018-05-08 14:55:02,410+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}' Regards, Callum [cid:672081BE-E695-46B2-965B-D34194DE1D60@well.ox.ac.uk][cid:030D44DD-3DED-412D-856D-E7A458E495AE@well.ox.ac.uk] -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 8 May 2018, at 14:50, Michael Burman <mburman@redhat.com<mailto:mburman@redhat.com>> wrote: Have you removed the required property? Do you have other required networks in this cluster except ovirtmgmt? Do you see event massages complaining that some network/s is missing in the cluster? What versions are your engine and vdsm? Alona, am i missing something? On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Dear Michael, No luck, the cycling is still happening. This is with a hard reset of the host-engine vm, the host, updating the node versions on everything. Would it be really bad to just delete the nodes from the DB and clean install them? Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 8 May 2018, at 13:44, Michael Burman <mburman@redhat.com<mailto:mburman@redhat.com>> wrote: You should only remove the required property from the network, this will release the host from the activating cycle in which it's stuck now. On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Dear Michael, Good to know - didn't know about that path to configure networks. What's the procedure to remove the failing hosts and re-install? I can't get them into maintenance mode to then install them. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 8 May 2018, at 12:37, Michael Burman <mburman@redhat.com<mailto:mburman@redhat.com>> wrote: Hi Callum Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388 Do you have a required network in your env(except ovirtmgmt)? try to uncheck the 'required' from this network, this will solve the issue(via Clusters>network or Networks>Cluster). The bug was fixed in ovirt-engine-4.2.3.4 Cheers) On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Dear All, There appears to be an issue with the host install on these, there's quite a lot of errors being kicked out into the logs as they cycle through attempting to activate, failing, and then failing to go into maintenance. I can't remove the hosts from the engine to attempt to reinstall them. Issue presented from having a statically configured network on the ovritmgmt network on the hosts before running the host install. There are SQL errors (FK missing) and assorted goodness in there. Any help greatly appreciated, not sure what the next step forward is from here Engine.log on dropbox: https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0 PS. Sorry if this comes through a few times, mailing list membership seems to be having a funny turn this morning. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> -- Michael Burman Senior Quality engineer - rhv network - redhat israel Red Hat <https://www.redhat.com/> mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725> IM: mburman [https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h... -- Michael Burman Senior Quality engineer - rhv network - redhat israel Red Hat <https://www.redhat.com/> mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725> IM: mburman [https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h... -- Michael Burman Senior Quality engineer - rhv network - redhat israel Red Hat <https://www.redhat.com/> mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725> IM: mburman [https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...