Run VM failure on hosted engine system test 4.0

Eyal Edri eedri at redhat.com
Tue Aug 16 08:25:09 UTC 2016 

FYI,

The run VM test is failing every few runs on the HE 4.0 system tests job
[1],
engine & hosts logs can be found here [2]
Seems like engine is loosing connection to the DB?

This test isn't failing on the basic suite.

2016-08-13 20:18:01,488 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-1) [7e13fc89] Error:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException:
Connection issues during send request
2016-08-13 20:18:01,488 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-1) [7e13fc89] Exception:
java.util.concurrent.ExecutionException:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException:
Connection issues during send request
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[rt.jar:1.8.0_101]
	at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_101]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:68)
[vdsbroker.jar:]
	at org.ovirt.engine.core.bll.transport.ProtocolDetector.attemptConnection(ProtocolDetector.java:74)
[bll.jar:]
	at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:190)
[bll.jar:]
	at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:90)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1301)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1443)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2071)
[bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166)
[utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105)
[utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1486)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398)
[bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:670)
[bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_101]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_101]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
[wildfly-weld-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
[wildfly-weld-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
[wildfly-weld-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
[weld-core-impl-2.3.2.Final.jar:2.3.2.Final]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
[wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
	at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view4.runInternalAction(Unknown
Source) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_101]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_101]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101]
	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:436)
[weld-core-impl-2.3.2.Final.jar:2.3.2.Final]
	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
[weld-core-impl-2.3.2.Final.jar:2.3.2.Final]
	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.3.2.Final.jar:2.3.2.Final]
	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.3.2.Final.jar:2.3.2.Final]
	at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2467)
[bll.jar:]
	at org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.lambda$executeCommand$2(AddVdsCommand.java:207)
[bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92)
[utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_101]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
Caused by: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException:
Connection issues during send request
	at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:95)
[vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundPoll$2(JsonRpcVdsServer.java:972)
[vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:458)
[vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:447)
[vdsbroker.jar:]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
	... 5 more
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
Connection failed
	at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:165)
[vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:134)
[vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:81)
[vdsm-jsonrpc-java-client.jar:]
	at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91)
[vdsbroker.jar:]
	... 9 more

2016-08-13 20:18:01,579 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-1) [7e13fc89]

\ Command 'TimeBoundPollVDSCommand(HostName =
lago_he_basic_suite_4_0_host0,
TimeBoundPollVDSCommandParameters:{runAsync='true',
hostId='a37975a5-9202-4e08-9eed-a2bd0f41db42'})'

\ execution failed: java.util.concurrent.ExecutionException:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException:
Connection issues during send request



[1] http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/139/
[2]
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/139/artifact/exported-artifacts/test_logs/he_basic_suite_4.0/post-004_basic_sanity.py/

-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160816/ba0c3751/attachment.html>

More information about the Infra mailing list