This is also appearing on 4.3 on test:  002_bootstrap.download_engine_certs
http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/654/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.3/post-002_bootstrap.py

Master patch failed on host deploy but I think it would fail on the same if I re-run.


On Sun, Apr 14, 2019 at 9:22 AM Dafna Ron <dron@redhat.com> wrote:
Hi,

We had a failure on both basic and upgrade suites.
The issue seems to be in initialize engine with the below error

Root cause change: https://gerrit.ovirt.org/#/c/99278/ - Purge old events and EventPublisher should handle exceptions

Logs can be found here:

error:

2019-04-12 03:36:25,571-04 ERROR [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 44) [] Error during initialization: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke private void org.ovirt.engine.core.vdsbroker.monitoring.VmMigrationProgressMonitoring.subscribe() on org.ovirt.engine.core.vdsbroker.monitoring.VmMigrationProgressMonitoring@c22d2ea
        at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:85) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:66) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:140) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:700) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.initialize(Backend.java:312) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.create(Backend.java:195) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:78)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:125)
        at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:111)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53) [wildfly-weld-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) [wildfly-weld-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.component.ComponentInstantiatorInterceptor.processInvocation(ComponentInstantiatorInterceptor.java:74)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.weld.interceptors.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:105)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:70) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) [wildfly-weld-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
        at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
        at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
       at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:127) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:141) [wildfly-ejb3-14.0.1.Final.jar:14.0.1.Final]
        at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_201]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_201]
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201]
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
        at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        ... 69 more
Caused by: java.lang.NoSuchMethodError: org.ovirt.vdsm.jsonrpc.client.reactors.ReactorFactory.getWorker(I)Lorg/ovirt/vdsm/jsonrpc/client/internal/ResponseWorker;
        at org.ovirt.engine.core.vdsbroker.ResourceManager.subscribe(ResourceManager.java:458) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.subscribe$$super(Unknown Source) [vdsbroker.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
        at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
        at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.subscribe(Unknown Source) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.VmMigrationProgressMonitoring.subscribe(VmMigrationProgressMonitoring.java:35) [vdsbroker.jar:]
        ... 74 more

2019-04-12 03:36:46,992-04 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-1) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US'
2019-04-12 03:36:47,025-04 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-1) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US'
2019-04-12 03:36:47,027-04 DEBUG [org.ovirt.engine.core.uutils.ssh.OpenSSHUtils] (default task-1) [] Exponent is 65537 (010001).
2019-04-12 03:36:47,027-04 DEBUG [org.ovirt.engine.core.uutils.ssh.OpenSSHUtils] (default task-1) [] Modulus is 23045455587502151351307881395746676489582617487444460434296505697360842634378900404903793405238811523302482543003054056200741987458008963931649820493880963560986441075262188689772713221069929429876133085702357731148902133770193496183003052020349333463043013537857088409307780071693644400706574082881179894409681722899845950989200033040393214136788457749540203592306287581368470141041245429472069137351244033201424938951360962572394606552585936959608039675862808386819256128109473508142431203907117802195732456069118309327167040542073925080491412451765973629566066094113148306929852301896069383522789589872834454425973 (010001).