10:49 a.m.
Hi,
Couple of days ago I had to refresh my dev environment to validate the issue and since
support for Fedora 28 is no longer there I used CentOS 7 for both engine and host.
Vdsm and cockpit socket on host is from the package, engine is compiled from git master.
As I try to add new host, installment proceeds for quite a while but fails in the end,
setting host to non operational. I see the following in the engine logs:
2019-08-12 17:32:40,341+02 INFO [org.ovirt.engine.core.bll.host.HostConnectivityChecker]
(EE-ManagedThreadFactory-engine-Thread-20) [9f80a7] Engine managed to communicate with
VDSM agent on host 'banana' with address 'banana'
('bcabbd09-81ef-422d-811c-3a18ac97919f')
2019-08-12 17:32:42,163+02 WARN [org.ovirt.engine.core.bll.network.NetworkConfigurator]
(EE-ManagedThreadFactory-engine-Thread-20) [9f80a7] Failed to find a valid interface for
the management network of host banana. If the interface null is a bridge, it should be
torn-down manually.
2019-08-12 17:32:42,166+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-20) [9f80a7] Exception:
org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException:
Interface null is invalid for management network
at
org.ovirt.engine.core.bll.network.NetworkConfigurator.findNicToSetupManagementNetwork(NetworkConfigurator.java:159)
[bll.jar:]
at
org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:82)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:291)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:203)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:112)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1326)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2005)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164)
[utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103)
[utils.jar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1386) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:420) [bll.jar:]
at
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:639) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source) [:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
[weld-ejb-3.1.1.Final.jar:3.1.1.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
[wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
Source) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown Source) [:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222]
at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410)
[weld-core-impl-3.1.1.Final.jar:3.1.1.Final]
at
org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134)
[weld-ejb-3.1.1.Final.jar:3.1.1.Final]
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-3.1.1.Final.jar:3.1.1.Final]
at
org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68)
[weld-ejb-3.1.1.Final.jar:3.1.1.Final]
at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106)
[weld-core-impl-3.1.1.Final.jar:3.1.1.Final]
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2395)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.lambda$executeCommand$3(AddVdsCommand.java:226)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96)
[utils.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_222]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222]
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
[javax.enterprise.concurrent-1.0.jar:]
2019-08-12 17:32:42,188+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-20) [9f80a7] Host installation failed for host
'bcabbd09-81ef-422d-811c-3a18ac97919f', 'banana': Failed to configure
management network on the host
...
2019-08-12 17:35:01,502+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-85) [3c4801f1] Host 'banana' is
set to Non-Operational, it is missing the following networks: 'ovirtmgmt'
On host, sudo systemctl status vdsmd shows vdsm as active (running) with two concerning
messages:
авг 12 17:30:01 banana vdsm[2309]: WARN MOM not available.
авг 12 17:30:01 banana vdsm[2309]: WARN MOM not available, KSM stats will be missing.
I found a similar bug, https://bugzilla.redhat.com/show_bug.cgi?id=1676695 but it was
fixed for 4.30 while on host I use 4.40.0-696
Have you seen such issue? Do you think networking error is related to MOM warnings?
Setup I use is very much what worked perfectly fine for fedora and networking-vise the
only questionable thing I am doing is having this kind of records in machine's
/etc/hosts files to use names instead IPs (as imageio can't stand certificates with
IPs):
192.168.111.2 apple
192.168.111.3 banana
But this can't be the reason, right?
Fedor