3:44 a.m.
After a reboot I can’t access the management ovirt-engine webpage anymore.
server.log line that looks bad:
2017-06-04 21:35:55,652-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0190: Step handler
org.jboss.as.server.deployment.DeploymentHandlerUtil$5@61b686ed for operation
{"operation" => "undeploy","address" =>
[("deployment" => "engine.ear")],"owner" =>
[("subsystem" => "deployment-scanner"),("scanner" =>
"default")]} at address [("deployment" => "engine.ear")]
failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345:
Timeout after 5 seconds waiting for existing service service
jboss.deployment.unit."engine.ear".contents to be removed so a new instance can
be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds
waiting for existing service service jboss.deployment.unit."engine.ear".contents
to be removed so a new instance can be installed.
at
org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:2107)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.PathContentServitor.addService(PathContentServitor.java:50)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.DeploymentHandlerUtil.doDeploy(DeploymentHandlerUtil.java:165)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.DeploymentHandlerUtil$5$1.handleResult(DeploymentHandlerUtil.java:333)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:756)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
at org.jboss.as.controller.ModelControllerImpl$3$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[jboss-threads-2.2.1.Final.jar:2.2.1.Final]
2017-06-04 21:36:00,654-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0349: Timeout after [5] seconds waiting for service
container stability while finalizing an operation. Process must be restarted. Step that
first updated the service container was 'undeploy' at address
'[("deployment" => "engine.ear")]'
2017-06-04 21:36:00,655-04 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 2)
WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service
jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START
Similar issue in engine.log:
2017-06-04 21:30:50,076-04 ERROR
[org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool --
40) [38484164] Failed to initialize backend: org.jboss.weld.exceptions.WeldException:
WELD-000049: Unable to invoke private void
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init() on
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller@438212cb
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34) [bll.jar:]
at
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:65)
[bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
at
org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
... 64 more
Caused by: java.lang.NullPointerException
at
org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.isDataOperationsByHSM(BaseImagesCommand.java:128)
[bll.jar:]
at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.isUsingSPDMFlow(CopyImageGroupCommand.java:105)
[bll.jar:]
at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.getCallback(CopyImageGroupCommand.java:188)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandsRepository.addToCallbackMap(CommandsRepository.java:69)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.initCommandExecutor(CommandCallbacksPoller.java:191)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init(CommandCallbacksPoller.java:41)
[bll.jar:]
... 69 more
8:24 a.m.
On Mon, Jun 5, 2017 at 4:44 AM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
After a reboot I can’t access the management ovirt-engine webpage
anymore.
server.log line that looks bad:
2017-06-04 21:35:55,652-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0190: Step handler
org.jboss.as.server.deployment.DeploymentHandlerUtil$5@61b686ed for operation
{"operation" => "undeploy","address" =>
[("deployment" => "engine.ear")],"owner" =>
[("subsystem" => "deployment-scanner"),("scanner" =>
"default")]} at address [("deployment" => "engine.ear")]
failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345:
Timeout after 5 seconds waiting for existing service service
jboss.deployment.unit."engine.ear".contents to be removed so a new instance can
be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds
waiting for existing service service jboss.deployment.unit."engine.ear".contents
to be removed so a new instance can be installed.
at
org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:2107)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.PathContentServitor.addService(PathContentServitor.java:50)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.DeploymentHandlerUtil.doDeploy(DeploymentHandlerUtil.java:165)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.server.deployment.DeploymentHandlerUtil$5$1.handleResult(DeploymentHandlerUtil.java:333)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:756)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
at
org.jboss.as.controller.ModelControllerImpl$3$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[jboss-threads-2.2.1.Final.jar:2.2.1.Final]
2017-06-04 21:36:00,654-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0349: Timeout after [5] seconds waiting for service
container stability while finalizing an operation. Process must be restarted. Step that
first updated the service container was 'undeploy' at address
'[("deployment" => "engine.ear")]'
2017-06-04 21:36:00,655-04 INFO [org.jboss.as.controller] (DeploymentScanner-threads -
2) WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service
jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START
Similar issue in engine.log:
2017-06-04 21:30:50,076-04 ERROR
[org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool --
40) [38484164] Failed to initialize backend: org.jboss.weld.exceptions.WeldException:
WELD-000049: Unable to invoke private void
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init() on
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller@438212cb
Seems somewhat similar to [1]. Sounds familiar? Adding Arik.
Best,
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1345853
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34)
[bll.jar:]
at
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:65)
[bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
at
org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
at
org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
at
org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
... 64 more
Caused by: java.lang.NullPointerException
at
org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.isDataOperationsByHSM(BaseImagesCommand.java:128)
[bll.jar:]
at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.isUsingSPDMFlow(CopyImageGroupCommand.java:105)
[bll.jar:]
at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.getCallback(CopyImageGroupCommand.java:188)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandsRepository.addToCallbackMap(CommandsRepository.java:69)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.initCommandExecutor(CommandCallbacksPoller.java:191)
[bll.jar:]
at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init(CommandCallbacksPoller.java:41)
[bll.jar:]
... 69 more
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Didi
11:12 a.m.
On Jun 5, 2017, at 2:24 AM, Yedidyah Bar David
<didi(a)redhat.com> wrote:
On Mon, Jun 5, 2017 at 4:44 AM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
> After a reboot I can’t access the management ovirt-engine webpage anymore.
>
>
> server.log line that looks bad:
>
> 2017-06-04 21:35:55,652-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0190: Step handler
org.jboss.as.server.deployment.DeploymentHandlerUtil$5@61b686ed for operation
{"operation" => "undeploy","address" =>
[("deployment" => "engine.ear")],"owner" =>
[("subsystem" => "deployment-scanner"),("scanner" =>
"default")]} at address [("deployment" => "engine.ear")]
failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345:
Timeout after 5 seconds waiting for existing service service
jboss.deployment.unit."engine.ear".contents to be removed so a new instance can
be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds
waiting for existing service service jboss.deployment.unit."engine.ear".contents
to be removed so a new instance can be installed.
> at
org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:2107)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.server.deployment.PathContentServitor.addService(PathContentServitor.java:50)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.server.deployment.DeploymentHandlerUtil.doDeploy(DeploymentHandlerUtil.java:165)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.server.deployment.DeploymentHandlerUtil$5$1.handleResult(DeploymentHandlerUtil.java:333)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:756)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at java.security.AccessController.doPrivileged(Native Method)
[rt.jar:1.8.0_131]
> at
org.jboss.as.controller.ModelControllerImpl$3$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[jboss-threads-2.2.1.Final.jar:2.2.1.Final]
>
> 2017-06-04 21:36:00,654-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0349: Timeout after [5] seconds waiting for service
container stability while finalizing an operation. Process must be restarted. Step that
first updated the service container was 'undeploy' at address
'[("deployment" => "engine.ear")]'
> 2017-06-04 21:36:00,655-04 INFO [org.jboss.as.controller] (DeploymentScanner-threads
- 2) WFLYCTL0183: Service status report
> WFLYCTL0186: Services which failed to start: service
jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START
>
>
>
> Similar issue in engine.log:
>
> 2017-06-04 21:30:50,076-04 ERROR
[org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool --
40) [38484164] Failed to initialize backend: org.jboss.weld.exceptions.WeldException:
WELD-000049: Unable to invoke private void
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init() on
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller@438212cb
Seems somewhat similar to [1]. Sounds familiar? Adding Arik.
Best,
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1345853
Seems possible, I was testing a new storage domain, and was adding and deleting it a
couple times with similar names. And a VM with similar names for the tests, Any thoughts
on how to recover?
> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34)
[bll.jar:]
> at
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:65)
[bll.jar:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
> at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
> at
org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
> at
org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
> ... 64 more
> Caused by: java.lang.NullPointerException
> at
org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.isDataOperationsByHSM(BaseImagesCommand.java:128)
[bll.jar:]
> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.isUsingSPDMFlow(CopyImageGroupCommand.java:105)
[bll.jar:]
> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.getCallback(CopyImageGroupCommand.java:188)
[bll.jar:]
> at
org.ovirt.engine.core.bll.tasks.CommandsRepository.addToCallbackMap(CommandsRepository.java:69)
[bll.jar:]
> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.initCommandExecutor(CommandCallbacksPoller.java:191)
[bll.jar:]
> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init(CommandCallbacksPoller.java:41)
[bll.jar:]
> ... 69 more
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
5:43 p.m.
As a follow-up for future reference. Re-running engine-setup with the same config file
solved all my issues. I can now long into the website, and run VM’s again.
On Jun 5, 2017, at 5:12 AM, Thomas Wakefield <dwakefi2(a)gmu.edu>
wrote:
> On Jun 5, 2017, at 2:24 AM, Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Mon, Jun 5, 2017 at 4:44 AM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
>> After a reboot I can’t access the management ovirt-engine webpage anymore.
>>
>>
>> server.log line that looks bad:
>>
>> 2017-06-04 21:35:55,652-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0190: Step handler
org.jboss.as.server.deployment.DeploymentHandlerUtil$5@61b686ed for operation
{"operation" => "undeploy","address" =>
[("deployment" => "engine.ear")],"owner" =>
[("subsystem" => "deployment-scanner"),("scanner" =>
"default")]} at address [("deployment" => "engine.ear")]
failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345:
Timeout after 5 seconds waiting for existing service service
jboss.deployment.unit."engine.ear".contents to be removed so a new instance can
be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds
waiting for existing service service jboss.deployment.unit."engine.ear".contents
to be removed so a new instance can be installed.
>> at
org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:2107)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.server.deployment.PathContentServitor.addService(PathContentServitor.java:50)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.server.deployment.DeploymentHandlerUtil.doDeploy(DeploymentHandlerUtil.java:165)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.server.deployment.DeploymentHandlerUtil$5$1.handleResult(DeploymentHandlerUtil.java:333)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:756)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at java.security.AccessController.doPrivileged(Native Method)
[rt.jar:1.8.0_131]
>> at
org.jboss.as.controller.ModelControllerImpl$3$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
>> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
>> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[jboss-threads-2.2.1.Final.jar:2.2.1.Final]
>>
>> 2017-06-04 21:36:00,654-04 ERROR [org.jboss.as.controller.management-operation]
(DeploymentScanner-threads - 2) WFLYCTL0349: Timeout after [5] seconds waiting for service
container stability while finalizing an operation. Process must be restarted. Step that
first updated the service container was 'undeploy' at address
'[("deployment" => "engine.ear")]'
>> 2017-06-04 21:36:00,655-04 INFO [org.jboss.as.controller]
(DeploymentScanner-threads - 2) WFLYCTL0183: Service status report
>> WFLYCTL0186: Services which failed to start: service
jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START
>>
>>
>>
>> Similar issue in engine.log:
>>
>> 2017-06-04 21:30:50,076-04 ERROR
[org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool --
40) [38484164] Failed to initialize backend: org.jboss.weld.exceptions.WeldException:
WELD-000049: Unable to invoke private void
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init() on
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller@438212cb
>
> Seems somewhat similar to [1]. Sounds familiar? Adding Arik.
>
> Best,
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1345853
Seems possible, I was testing a new storage domain, and was adding and deleting it a
couple times with similar names. And a VM with similar names for the tests, Any thoughts
on how to recover?
>
>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:65)
[bll.jar:]
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
>> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
>> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
>> at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
>> at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
>> at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
>> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
>> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
>> at
org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
>> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at
org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
>> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
>> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
>> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
>> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
>> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>> ... 64 more
>> Caused by: java.lang.NullPointerException
>> at
org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.isDataOperationsByHSM(BaseImagesCommand.java:128)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.isUsingSPDMFlow(CopyImageGroupCommand.java:105)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.getCallback(CopyImageGroupCommand.java:188)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.tasks.CommandsRepository.addToCallbackMap(CommandsRepository.java:69)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.initCommandExecutor(CommandCallbacksPoller.java:191)
[bll.jar:]
>> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init(CommandCallbacksPoller.java:41)
[bll.jar:]
>> ... 69 more
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> Didi
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:16 a.m.
On Sat, Jun 10, 2017 at 6:43 PM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
As a follow-up for future reference. Re-running engine-setup with
the same config file solved all my issues. I can now long into the website, and run VM’s
again.
"same config file" - the one passed to --config-append=?
Interesting. It might be worthwhile to compare the setup logs created by
these runs. Perhaps you can open a BZ bug, describe your flow, and attach
the conf and log files?
Thanks for the report!
> On Jun 5, 2017, at 5:12 AM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
>
>
>> On Jun 5, 2017, at 2:24 AM, Yedidyah Bar David <didi(a)redhat.com> wrote:
>>
>> On Mon, Jun 5, 2017 at 4:44 AM, Thomas Wakefield <dwakefi2(a)gmu.edu> wrote:
>>> After a reboot I can’t access the management ovirt-engine webpage anymore.
>>>
>>>
>>> server.log line that looks bad:
>>>
>>> 2017-06-04 21:35:55,652-04 ERROR
[org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2)
WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$5@61b686ed
for operation {"operation" => "undeploy","address" =>
[("deployment" => "engine.ear")],"owner" =>
[("subsystem" => "deployment-scanner"),("scanner" =>
"default")]} at address [("deployment" => "engine.ear")]
failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345:
Timeout after 5 seconds waiting for existing service service
jboss.deployment.unit."engine.ear".contents to be removed so a new instance can
be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds
waiting for existing service service jboss.deployment.unit."engine.ear".contents
to be removed so a new instance can be installed.
>>> at
org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:2107)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.server.deployment.PathContentServitor.addService(PathContentServitor.java:50)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.server.deployment.DeploymentHandlerUtil.doDeploy(DeploymentHandlerUtil.java:165)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.server.deployment.DeploymentHandlerUtil$5$1.handleResult(DeploymentHandlerUtil.java:333)
[wildfly-server-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:644)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:756)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
org.jboss.as.controller.ModelControllerImpl$3$1$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at java.security.AccessController.doPrivileged(Native Method)
[rt.jar:1.8.0_131]
>>> at
org.jboss.as.controller.ModelControllerImpl$3$1.run(ModelControllerImpl.java:750)
[wildfly-controller-2.2.0.Final.jar:2.2.0.Final]
>>> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
>>> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
>>> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
[jboss-threads-2.2.1.Final.jar:2.2.1.Final]
>>>
>>> 2017-06-04 21:36:00,654-04 ERROR
[org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2)
WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while
finalizing an operation. Process must be restarted. Step that first updated the service
container was 'undeploy' at address '[("deployment" =>
"engine.ear")]'
>>> 2017-06-04 21:36:00,655-04 INFO [org.jboss.as.controller]
(DeploymentScanner-threads - 2) WFLYCTL0183: Service status report
>>> WFLYCTL0186: Services which failed to start: service
jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START
>>>
>>>
>>>
>>> Similar issue in engine.log:
>>>
>>> 2017-06-04 21:30:50,076-04 ERROR
[org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool --
40) [38484164] Failed to initialize backend: org.jboss.weld.exceptions.WeldException:
WELD-000049: Unable to invoke private void
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init() on
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller@438212cb
>>
>> Seems somewhat similar to [1]. Sounds familiar? Adding Arik.
>>
>> Best,
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1345853
>
> Seems possible, I was testing a new storage domain, and was adding and deleting it a
couple times with similar names. And a VM with similar names for the tests, Any thoughts
on how to recover?
>
>
>
>>
>>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:65)
[bll.jar:]
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
>>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
>>> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
>>> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
>>> at
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
>>> at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
>>> at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
>>> at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>>> at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
>>> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
>>> at
org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
>>> at
org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
>>> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>>> at
org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
>>> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_131]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
>>> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
>>> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
>>> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_131]
>>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_131]
>>> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_131]
>>> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_131]
>>> at
org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98)
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
>>> ... 64 more
>>> Caused by: java.lang.NullPointerException
>>> at
org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.isDataOperationsByHSM(BaseImagesCommand.java:128)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.isUsingSPDMFlow(CopyImageGroupCommand.java:105)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand.getCallback(CopyImageGroupCommand.java:188)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.tasks.CommandsRepository.addToCallbackMap(CommandsRepository.java:69)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.initCommandExecutor(CommandCallbacksPoller.java:191)
[bll.jar:]
>>> at
org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.init(CommandCallbacksPoller.java:41)
[bll.jar:]
>>> ... 69 more
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> --
>> Didi
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Didi
2532
days inactive
2538
days old
4 comments
2 participants
participants (2)
-
Thomas Wakefield -
Yedidyah Bar David