[ovirt-devel] engine startup failure: Backend has been recursively invoked

[Yevgeny Zaspitsky]

As you might guess on your own, the update sql is a temporary measure... 

The engine patch was published for the problem: https://gerrit.ovirt.org/#/c/43831/ 

Regards, 
Yevgeny 

----- Original Message -----

> From: "Marcin Mirecki" <mmirecki at redhat.com>
> To: devel at ovirt.org
> Sent: Monday, July 20, 2015 2:39:37 PM
> Subject: Re: [ovirt-devel] engine startup failure: Backend has been
> recursively invoked

> SOLVED (by Yevgeny)

> The error shows when a host is non-responsive before the engine is shut down.
> https://bugzilla.redhat.com/show_bug.cgi?id=1244718

> To get rid of the problem:
> UPDATE vds_dynamic SET status = 3 WHERE status = 4;

> ----- Original Message -----
> From: "Marcin Mirecki" <mmirecki at redhat.com>
> To: devel at ovirt.org
> Sent: Monday, July 20, 2015 9:30:04 AM
> Subject: [ovirt-devel] engine startup failure: Backend has been recursively
> invoked

> After pulling the engine today morning, it refuses to start with the
> following exception.
> Any clues what cause of the problem is?

> 2015-07-20 09:13:32,273 ERROR [org.ovirt.engine.core.bll.Backend]
> (ServerService Thread Pool -- 47) [] Error during initialization:
> org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke
> private void org.ovirt.engine.core.vdsbroker.ResourceManager.init() on
> org.ovirt.engine.core.vdsbroker.ResourceManager at 5576f8fd
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.BeanInjectionTarget.postConstruct(BeanInjectionTarget.java:70)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:153)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:840)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:370)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:381)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:121)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:150)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:762)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:168)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:88)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at org.ovirt.engine.core.bll.Backend.loadService(Backend.java:283) [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.initialize(Backend.java:271) [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.create(Backend.java:145) [bll.jar:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_45]
> at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_45]
> at
> org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
> at
> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:117)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
> at
> org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:55)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ee.component.ComponentInstantiatorInterceptor.processInvocation(ComponentInstantiatorInterceptor.java:75)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:94)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:260)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:399)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
> [wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> [wildfly-ee-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
> at
> org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at
> org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:160)
> at
> org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:133)
> at
> org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:89)
> at
> org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:137)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [rt.jar:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [rt.jar:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [rt.jar:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_45]
> at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_45]
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> ... 82 more
> Caused by: javax.ejb.EJBException: java.lang.IllegalStateException:
> JBAS014232: @PostConstruct method of EJB singleton Backend of type
> org.ovirt.engine.core.bll.Backend has been recursively invoked
> at
> org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:216)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:262)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:399)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:95)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
> at
> org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:448)
> at
> org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
> at
> org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)
> at
> org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at
> org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
> at
> org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.getResourceManager(Unknown
> Source) [common.jar:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_45]
> at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_45]
> at
> org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:414)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:65)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
> [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
> at
> org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.getResourceManager(Unknown
> Source) [bll.jar:]
> at
> org.ovirt.engine.core.bll.VdsEventListener.removeAsyncRunningCommand(VdsEventListener.java:490)
> [bll.jar:]
> at
> org.ovirt.engine.core.vdsbroker.ResourceManager.RemoveAsyncRunningVm(ResourceManager.java:202)
> [vdsbroker.jar:]
> at
> org.ovirt.engine.core.vdsbroker.ResourceManager.SetVmUnknown(ResourceManager.java:319)
> [vdsbroker.jar:]
> at
> org.ovirt.engine.core.vdsbroker.ResourceManager.init(ResourceManager.java:156)
> [vdsbroker.jar:]
> ... 87 more
> Caused by: java.lang.IllegalStateException: JBAS014232: @PostConstruct method
> of EJB singleton Backend of type org.ovirt.engine.core.bll.Backend has been
> recursively invoked
> at
> org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:119)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:47)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> at
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
> at
> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:260)
> [wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
> ... 132 more
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20150720/0fd13f90/attachment-0001.html>