I've attached a couple of screenshots, the hosts seem to not succeed in going into non-operational mode. Required network is now unticked. Storage is definitely accessible by all nodes + engine.

The moment they succeed into going non-operational, they cycle back to activating.

It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` = 'virtA003.cluster'; 

2018-05-08 14:54:32,351+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E
E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 313cfe33
2018-05-08 14:54:32,353+01 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory
-engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:54:32,385+01 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact
ory-engineScheduled-Thread-30) [32499b7b] Running command: SetNonOperationalVdsCommand internal: true. Entitie
s affected :  ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS
2018-05-08 14:54:32,388+01 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac
tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName = virtA002.cluster, SetVdsSt
atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7', status='NonOperational', nonOperation
alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b5138fd
2018-05-08 14:54:32,705+01 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFac
tory-engineScheduled-Thread-36) [3c65dca0] FINISH, SetVdsStatusVDSCommand, log id: 44441e64
2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFact
ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set to Non-Operational, it is missing the
 following networks: 'ovirtmgmt'
2018-05-08 14:54:32,740+01 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the following networks are missing on host:
 'ovirtmgmt'
2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler] (EE-ManagedThreadFactory-eng
ineScheduled-Thread-36) [3c65dca0] Exception: org.springframework.jdbc.CannotGetJdbcConnectionException: Could
 not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ00045
7: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxCo
nnectionListener@769d34fc[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnec
tion@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=15257
87372702 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=Semaphor
eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@3aa2
9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQ
L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-
jdbc.jar:4.3.9.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619) [spring-jdbc.jar:4.3.9.RE
LEASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684) [spring-jdbc.jar:4.3.9.RELE
ASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716) [spring-jdbc.jar:4.3.9.RELE
ASE]
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766) [spring-jdbc.jar:4.3.9.RELE
ASE]
        at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallIntern
al(PostgresDbEngineDialect.java:152) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(Postgre
sDbEngineDialect.java:118) [dal.jar:]
        at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.j
ar:4.3.9.RELEASE]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:1
35) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.ja
va:105) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:9
7) [dal.jar:]
        at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149) [dal.jar:]
        at org.ovirt.engine.core.bll.job.ExecutionHandler.checkIfJobHasTasks(ExecutionHandler.java:893) [bll.j
ar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400) [bll.jar:]
        at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto
r.java:13) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0
_161]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterce
ptor.java:78)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterce
ptor.java:88)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsIntercepto
r.java:101)
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFa
ctory.java:63)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionT
imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextIntercep
tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDi
spatcherInterceptor.java:52)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvoc
ation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255) [wildfly-ejb3-11.
0.0.Final.jar:11.0.0.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381) [wildfly-ejb3-11.0.0.Fina
l.jar:11.0.0.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11
.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScop
eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
        at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivat
ionInterceptor.java:89)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(Curr
entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterce
ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.
java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInter
ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInt
erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.ja
va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor
.java:50)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.
java:60)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
        at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
        at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
        at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll
.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0
_161]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
        at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
        at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.
java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
        at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
 [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
        at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPoi
ntPropagatingEnterpriseTargetBeanInstance.java:67) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
        at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100) [weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
        at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Prox
y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
        at org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303) [bll.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.setNonOperational(HostNe
tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.enforceNetworkCompliance
(HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce
NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99) [vdsbroker.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSuppo
rt.java:202) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.jav
a:137) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:1
05) [utils.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.processRefreshCapabilitiesResponse(VdsManager.java:794)
[vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$BeforeFirstRefreshTreatmentCallback.onRes
ponse(HostMonitoring.java:767) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC
allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45) [vdsbroker.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$processResponse$1(JsonRpcClient.java:182) [vdsm-
jsonrpc-java-client.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_161]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161]
        at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141) [jav
ax.enterprise.concurrent-1.0.jar:]
        at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:]
        at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.run(ManagedScheduledThreadPoolExecutor.java:532) [javax.enterprise.concurrent-1.0.jar:]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
        at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactory
Impl.java:250) [javax.enterprise.concurrent-1.0.jar:]
        at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronMana
gedThreadFactory.java:78)
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedCo
nnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NOR
MAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0
lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.j
boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueueManagedConnec
tionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co
nnectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=jav
a:/ENGINEDataSource] txSync=null]
        at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
        at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.jav
a:64)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spri
ng-jdbc.jar:4.3.9.RELEASE]
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-
jdbc.jar:4.3.9.RELEASE]
        ... 107 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() c
l=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NORMAL managed connection=
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0 lastReturned=1525787672
740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false pool=org.jboss.jca.core.connectio
nmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=E
NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc connectionManager=5d6abc6
c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] tx
Sync=null]
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractC
onnectionManager.java:975)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectio
nManager.java:792)
        at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
        ... 110 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:561)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:970)
        ... 112 more
Caused by: java.lang.IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK
        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:296)
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:554)
        ... 113 more

2018-05-08 14:54:32,749+01 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] transaction rolled back
2018-05-08 14:54:32,749+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException: Transaction Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK
2018-05-08 14:54:32,749+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock acquired, from now a monitoring of host will be skipped for host 'virtA003.cluster' from data-center 'Default'
2018-05-08 14:54:32,768+01 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 635dd4c4
2018-05-08 14:54:32,771+01 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH, SetVdsStatusVDSCommand, log id: 635dd4c4
2018-05-08 14:54:32,774+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate host finished. Lock released. Monitoring can run now for host 'virtA003.cluster' from data-center 'Default'
2018-05-08 14:54:32,775+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed to object 'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]', sharedLocks=''}'
2018-05-08 14:54:35,736+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START, GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster, VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log id: 710291f0
2018-05-08 14:54:35,736+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH, GetHardwareInfoAsyncVDSCommand, log id: 710291f0
2018-05-08 14:54:35,774+01 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS
2018-05-08 14:54:35,776+01 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START, SetVdsStatusVDSCommand(HostName = virtA003.cluster, SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d', status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 707d7be2
2018-05-08 14:55:00,005+01 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1 hosts
2018-05-08 14:55:00,005+01 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering hosts id: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster
2018-05-08 14:55:00,042+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]', sharedLocks=''}'
2018-05-08 14:55:00,056+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running command: ActivateVdsCommand internal: true. Entities affected :  ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2018-05-08 14:55:00,056+01 INFO  [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before acquiring lock in order to prevent monitoring for host 'virtA002.cluster' from data-center 'Default'
2018-05-08 14:55:00,056+01 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:55:02,410+01 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}'

Regards,
Callum
--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 8 May 2018, at 14:50, Michael Burman <mburman@redhat.com> wrote:

Have you removed the required property?
Do you have other required networks in this cluster except ovirtmgmt?
Do you see event massages complaining that some network/s is missing in the cluster?
What versions are your engine and vdsm?

Alona, am i missing something?



On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear Michael,

No luck, the cycling is still happening. This is with a hard reset of the host-engine vm, the host, updating the node versions on everything.

Would it be really bad to just delete the nodes from the DB and clean install them?

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 8 May 2018, at 13:44, Michael Burman <mburman@redhat.com> wrote:

You should only remove the required property from the network, this will release the host from the activating cycle in which it's stuck now.

On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear Michael,

Good to know - didn't know about that path to configure networks.

What's the procedure to remove the failing hosts and re-install? I can't get them into maintenance mode to then install them.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 8 May 2018, at 12:37, Michael Burman <mburman@redhat.com> wrote:

Hi Callum
Looks like you hitting  https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to uncheck the 'required' from this network, this will solve the issue(via Clusters>network or Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4

Cheers)

On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,

There appears to be an issue with the host install on these, there's quite a lot of errors being kicked out into the logs as they cycle through attempting to activate, failing, and then failing to go into maintenance. I can't remove the hosts from the engine to attempt to reinstall them.

Issue presented from having a statically configured network on the ovritmgmt network on the hosts before running the host install. There are SQL errors (FK missing) and assorted goodness in there.

Any help greatly appreciated, not sure what the next step forward is from here

Engine.log on dropbox:

PS. Sorry if this comes through a few times, mailing list membership seems to be having a funny turn this morning.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org




--
Michael Burman

Senior Quality engineer - rhv network - redhat israel

mburman@redhat.com    M: 0545355725     IM: mburman




--
Michael Burman

Senior Quality engineer - rhv network - redhat israel

mburman@redhat.com    M: 0545355725     IM: mburman




--
Michael Burman

Senior Quality engineer - rhv network - redhat israel

mburman@redhat.com    M: 0545355725     IM: mburman