[ovirt-users] Re: Host install failed - cannot set maintenance or remove

I've attached a couple of screenshots, the hosts seem to not succeed in going into non-operational mode. Required network is now unticked. Storage is definitely accessible by all nodes + engine. The moment they succeed into going non-operational, they cycle back to activating. It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` = 'virtA003.cluster'; 2018-05-08 14:54:32,351+01 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 313cfe33 2018-05-08 14:54:32,353+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory -engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[ 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}' 2018-05-08 14:54:32,385+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact ory-engineScheduled-Thread-30) [32499b7b] Running command: SetNonOperationalVdsCommand internal: true. Entitie s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS 2018-05-08 14:54:32,388+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName = virtA002.cluster, SetVdsSt atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7', status='NonOperational', nonOperation alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b5138fd 2018-05-08 14:54:32,705+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac tory-engineScheduled-Thread-36) [3c65dca0] FINISH, SetVdsStatusVDSCommand, log id: 44441e64 2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set to Non-Operational, it is missing the following networks: 'ovirtmgmt' 2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-Ma nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host v irtA003.cluster does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' 2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler] (EE-ManagedThreadFactory-eng ineScheduled-Thread-36) [3c65dca0] Exception: org.springframework.jdbc. CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ00045 7: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxCo nnectionListener@769d34fc[state=NORMAL managed connection=org.jboss.jca. adapters.jdbc.local.LocalManagedConnec tion@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=15257 87372702 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool. strategy.OnePool@55129ed4 mcp=Semaphor eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@3aa2 9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQ L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] at org.springframework.jdbc.datasource.DataSourceUtils. getConnection(DataSourceUtils.java:80) [spring- jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619) [spring-jdbc.jar:4.3.9.RE LEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684) [spring-jdbc.jar:4.3.9.RELE ASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716) [spring-jdbc.jar:4.3.9.RELE ASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766) [spring-jdbc.jar:4.3.9.RELE ASE] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$ PostgresSimpleJdbcCall.executeCallIntern al(PostgresDbEngineDialect.java:152) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$ PostgresSimpleJdbcCall.doExecute(Postgre sDbEngineDialect.java:118) [dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.j ar:4.3.9.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler. executeImpl(SimpleJdbcCallsHandler.java:1 35) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler. executeReadList(SimpleJdbcCallsHandler.ja va:105) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler. executeRead(SimpleJdbcCallsHandler.java:9 7) [dal.jar:] at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149) [dal.jar:] at org.ovirt.engine.core.bll.job.ExecutionHandler. checkIfJobHasTasks(ExecutionHandler.java:893) [bll.j ar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400) [bll.jar:] at org.ovirt.engine.core.bll.executor. DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto r.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0 _161] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161] at org.jboss.as.ee.component.ManagedReferenceMethodIntercep tor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. delegateInterception(Jsr299BindingsInterce ptor.java:78) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. doMethodInterception(Jsr299BindingsInterce ptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. processInvocation(Jsr299BindingsIntercepto r.java:101) at org.jboss.as.ee.component.interceptors. UserInterceptorFactory$1.processInvocation(UserInterceptorFa ctory.java:63) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics. ExecutionTimeInterceptor.processInvocation(ExecutionT imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor. processInvocation(ConcurrentContextIntercep tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation( InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors. ComponentDispatcherInterceptor.processInvocation(ComponentDi spatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton. SingletonComponentInstanceAssociationInterceptor.processInvoc ation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255) [wildfly-ejb3-11. 0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381) [wildfly-ejb3-11.0.0.Fina l.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11 .0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat ionInterceptor.aroundInvoke(AbstractEJBRequestScop eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final. jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter ceptor.processInvocation(EjbRequestScopeActivat ionInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. CurrentInvocationContextInterceptor.processInvocation(Curr entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final. jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics. WaitTimeInterceptor.processInvocation(WaitTimeInterce ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor. processInvocation(SecurityContextInterceptor. java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors. StartupAwaitInterceptor.processInvocation(StartupAwaitInter ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. ShutDownInterceptorFactory$1.processInvocation(ShutDownInt erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor. processInvocation(LoggingInterceptor.ja va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor. processInvocation(NamespaceContextInterceptor .java:50) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor. processInvocation(ContextClassLoaderInterceptor. java:60) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor. processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke( ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation( ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke( ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$ view3.runInternalAction(Unknown Source) [bll .jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0 _161] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161] at org.jboss.weld.util.reflection.Reflections. invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2. 4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandl er.invoke(EnterpriseBeanProxyMethodHandler. java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke( EnterpriseTargetBeanInstance.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnter priseTargetBeanInstance.invoke(InjectionPoi ntPropagatingEnterpriseTargetBeanInstance.java:67) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2. 4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$ BackendInternal$BackendLocal$2049259618$Prox y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303) [bll.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. HostNetworkTopologyPersisterImpl.setNonOperational(HostNe tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. HostNetworkTopologyPersisterImpl.enforceNetworkCompliance (HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99) [vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInNewTransaction(TransactionSuppo rt.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInRequired(TransactionSupport.jav a:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInScope(TransactionSupport.java:1 05) [utils.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager. processRefreshCapabilitiesResponse(VdsManager.java:794) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$ BeforeFirstRefreshTreatmentCallback.onRes ponse(HostMonitoring.java:767) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45) [vdsbroker.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$ processResponse$1(JsonRpcClient.java:182) [vdsm- jsonrpc-java-client.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_161] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161] at org.glassfish.enterprise.concurrent.internal. ManagedFutureTask.run(ManagedFutureTask.java:141) [jav ax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal. ManagedScheduledThreadPoolExecutor$ManagedScheduledFut ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal. ManagedScheduledThreadPoolExecutor$ManagedScheduledFut ureTask.run(ManagedScheduledThreadPoolExecutor.java:532) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ ManagedThread.run(ManagedThreadFactory Impl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ ElytronManagedThread.run(ElytronMana gedThreadFactory.java:78) Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedCo nnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener. TxConnectionListener@769d34fc[state=NOR MAL managed connection=org.jboss.jca.adapters.jdbc.local. LocalManagedConnection@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.j boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp= SemaphoreConcurrentLinkedQueueManagedConnec tionPool@78b9d77[pool=ENGINEDataSource] xaResource= LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co nnectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=jav a:/ENGINEDataSource] txSync=null] at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection( WrapperDataSource.java:146) at org.jboss.as.connector.subsystems.datasources. WildFlyDataSource.getConnection(WildFlyDataSource.jav a:64) at org.springframework.jdbc.datasource.DataSourceUtils. doGetConnection(DataSourceUtils.java:111) [spri ng-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils. getConnection(DataSourceUtils.java:77) [spring- jdbc.jar:4.3.9.RELEASE] ... 107 more Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() c l=org.jboss.jca.core.connectionmanager.listener. TxConnectionListener@769d34fc[state=NORMAL managed connection= org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0 lastReturned=1525787672 740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.jboss.jca.core.connectio nmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueue ManagedConnectionPool@78b9d77[pool=E NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc connectionManager=5d6abc6 c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] tx Sync=null] at org.jboss.jca.core.connectionmanager.AbstractConnectionManager. reconnectManagedConnection(AbstractC onnectionManager.java:975) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager. allocateConnection(AbstractConnectio nManager.java:792) at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection( WrapperDataSource.java:138) ... 110 more Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object at org.jboss.jca.core.connectionmanager.tx. TxConnectionManagerImpl.managedConnectionReconnected( TxConnectionManagerImpl.java:561) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager. reconnectManagedConnection(AbstractConnectionManager.java:970) ... 112 more Caused by: java.lang.IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK at org.jboss.jca.core.connectionmanager.listener. TxConnectionListener.enlist(TxConnectionListener.java:296) at org.jboss.jca.core.connectionmanager.tx. TxConnectionManagerImpl.managedConnectionReconnected( TxConnectionManagerImpl.java:554) ... 113 more 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] transaction rolled back 2018-05-08 14:54:32,749+01 ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock acquired, from now a monitoring of host will be skipped for host 'virtA003.cluster' from data-center 'Default' 2018-05-08 14:54:32,768+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 635dd4c4 2018-05-08 14:54:32,771+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH, SetVdsStatusVDSCommand, log id: 635dd4c4 2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate host finished. Lock released. Monitoring can run now for host 'virtA003.cluster' from data-center 'Default' 2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed to object 'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]', sharedLocks=''}' 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START, GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster, VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log id: 710291f0 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 710291f0 2018-05-08 14:54:35,774+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS 2018-05-08 14:54:35,776+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 707d7be2 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1 hosts 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering hosts id: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster 2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]', sharedLocks=''}' 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running command: ActivateVdsCommand internal: true. Entities affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before acquiring lock in order to prevent monitoring for host 'virtA002.cluster' from data-center 'Default' 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e- b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}' 2018-05-08 14:55:02,410+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e- b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}' Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum(a)well.ox.ac.uk On 8 May 2018, at 14:50, Michael Burman <mburman(a)redhat.com&gt; wrote: Have you removed the required property? Do you have other required networks in this cluster except ovirtmgmt? Do you see event massages complaining that some network/s is missing in the cluster? What versions are your engine and vdsm? Alona, am i missing something? On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum(a)well.ox.ac.uk&gt; wrote: > Dear Michael, > > No luck, the cycling is still happening. This is with a hard reset of the > host-engine vm, the host, updating the node versions on everything. > > Would it be really bad to just delete the nodes from the DB and clean > install them? > > Regards, > Callum > > -- > > Callum Smith > Research Computing Core > Wellcome Trust Centre for Human Genetics > University of Oxford > e. callum(a)well.ox.ac.uk > > On 8 May 2018, at 13:44, Michael Burman <mburman(a)redhat.com&gt; wrote: > > You should only remove the required property from the network, this will > release the host from the activating cycle in which it's stuck now. > > On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum(a)well.ox.ac.uk&gt; > wrote: > >> Dear Michael, >> >> Good to know - didn't know about that path to configure networks. >> >> What's the procedure to remove the failing hosts and re-install? I can't >> get them into maintenance mode to then install them. >> >> Regards, >> Callum >> >> -- >> >> Callum Smith >> Research Computing Core >> Wellcome Trust Centre for Human Genetics >> University of Oxford >> e. callum(a)well.ox.ac.uk >> >> On 8 May 2018, at 12:37, Michael Burman <mburman(a)redhat.com&gt; wrote: >> >> Hi Callum >> Looks like you hitting https://bugzilla.redhat.com/sh >> ow_bug.cgi?id=1570388 >> Do you have a required network in your env(except ovirtmgmt)? try to >> uncheck the 'required' from this network, this will solve the issue(via >> Clusters>network or Networks>Cluster). >> The bug was fixed in ovirt-engine-4.2.3.4 >> >> Cheers) >> >> On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk&gt; >> wrote: >> >>> Dear All, >>> >>> There appears to be an issue with the host install on these, there's >>> quite a lot of errors being kicked out into the logs as they cycle through >>> attempting to activate, failing, and then failing to go into maintenance. I >>> can't remove the hosts from the engine to attempt to reinstall them. >>> >>> Issue presented from having a statically configured network on the >>> ovritmgmt network on the hosts before running the host install. There are >>> SQL errors (FK missing) and assorted goodness in there. >>> >>> Any help greatly appreciated, not sure what the next step forward is >>> from here >>> >>> Engine.log on dropbox: >>> https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0 >>> >>> PS. Sorry if this comes through a few times, mailing list membership >>> seems to be having a funny turn this morning. >>> >>> Regards, >>> Callum >>> >>> -- >>> >>> Callum Smith >>> Research Computing Core >>> Wellcome Trust Centre for Human Genetics >>> University of Oxford >>> e. callum(a)well.ox.ac.uk >>> >>> >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> >>> >> >> >> -- >> Michael Burman >> >> Senior Quality engineer - rhv network - redhat israel >> Red Hat >> >> <https://www.redhat.com/> >> >> mburman(a)redhat.com M: 0545355725 IM: mburman >> <https://red.ht/sig> >> >> >> > > > -- > Michael Burman > > Senior Quality engineer - rhv network - redhat israel > Red Hat > > <https://www.redhat.com/> > > mburman(a)redhat.com M: 0545355725 IM: mburman > <https://red.ht/sig> > > > -- Michael Burman Senior Quality engineer - rhv network - redhat israel Red Hat <https://www.redhat.com/> mburman(a)redhat.com M: 0545355725 IM: mburman <https://red.ht/sig>