Re: [ovirt-devel] System tests for 4.1 currently failing to run VMs!

--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: <mailto:eedri@redhat.com>> wrote: complicated than just build artifacts. I'm not saving we won't do it, = but it won't be ready tomorrow and we designed lago exactly for being = independent of any ci system so anyone can run it on their laptop.=20 Almost there. verification just because such a job doesn't exist yet.=20 <mailto:oourfali@redhat.com>> wrote: remember. Even after years I always keep asking is it "Re-run hooks=E2=80=9D= or =E2=80=9CRerun-Hooks=E2=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does = case matter=E2=80=A6..bleh comments (that would actually work good enough) <mailto:eedri@redhat.com>> wrote: <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: <mailto:ykaul@redhat.com>> wrote: <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote: <mailto:eedri@redhat.com>> wrote: <vfeenstr(a)redhat.com <mailto:vfeenstr@redhat.com>> wrote: <bkorren(a)redhat.com <mailto:bkorren@redhat.com>> wrote: <https://gerrit.ovirt.org/#/c/68826/> while running is unrelated. (vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') The vm start process = failed (vm:615) Traceback (most recent call last): File = "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm = self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run = self._connection.createXML(domxml, flags), File = "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, = in wrapper ret =3D f(*args, **kwargs) File = "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper = return func(inst, *args, **kwargs) File = "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML = if ret is None:raise libvirtError('virDomainCreateXML() failed', = conn=3Dself) libvirtError: internal error: process exited while = connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: = CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 = 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to = maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z = qemu-kvm: -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: = speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) = to bus "usb.0", port "1" (high speed)=20 prefferred point of view:) particular problem, so we can start with that and figure out the proper = approach later I think it's a better option. of large patch series combined with late detection it really is quite = risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out the = right revert patch is more complicated then fixing it. be early detection and not late detection?=20 issues in running it on a patch. maintainer who merged that patch ASAP, as that person is in the best = position to asses if revert is safe or if there is a simple follow up = patch he can push right away point to that particular patch in your email? It should start failing on = 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 = 01:56:07 2016 -0500) c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 = 15:16:40 2016 -0500) (vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: = internal error: process exited while connecting to monitor: = 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any = NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 = 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus = should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: = -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: speed = mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus = "usb.0", port "1" (high speed) (code=3D1) (vm:1197) 2016-12-20 = 16:54:43,550 INFO (vm/d299ab29) [virt.vm] = (vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Stopping connection = (guestagent:430)=20 [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command = 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: = VDSGenericException: VDSNetworkException: Timeout during rpc call = 2016-12-20 16:53:57,849-05 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp = Reactor) [7971dfb4] MESSAGE content-length:80 = destination:jms.topic.vdsm_responses content-type:application/json = subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", = "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}=EF=BF=BD = 2016-12-20 16:53:57,850-05 DEBUG = [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) = [] Message received: {"jsonrpc": "2.0", "id": = "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 = 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] = (ResponseWorker) [] Not able to update response for = "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Timeout during rpc call at = org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo= mmand.java:73) [vdsbroker.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos= tSetupNetworkPoller.java:56) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet= upNetworkPoller.java:41) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetu= pNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCom= mand(HostSetupNetworksCommand.java:287) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas= e.java:1249) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm= andBase.java:1389) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20= 53) [bll.jar:] at org.ovirt.engine.core.utils.tr = <http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.ex... teInSuppressed(TransactionSupport.java:164) [utils.jar:] at = org.ovirt.engine.core.utils.tr = <http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.ex... teInScope(TransactionSupport.java:103) [utils.jar:] at = org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) = [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) = [bll.jar:] at = org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(De= faultBackendActionExecutor.java:13) [bll.jar:] at = org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] = at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) = [bll.jar:] at = org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] = at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) = [:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat= ion(ManagedReferenceMethodInterceptor.java:52) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29= 9BindingsInterceptor.java:70) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29= 9BindingsInterceptor.java:80) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi= ndingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] = at = org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv= ocation(UserInterceptorFactory.java:63) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aro= undInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at = sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) = [:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.proce= ssInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.in = <http://org.jboss.as.ejb3.component.in/>vocationmetrics.ExecutionTimeIn... ceptor.processInvocation(ExecutionTimeInterceptor.java:43) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo= ke(AbstractEJBRequestScopeActivationInterceptor.java:73) = [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at = org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati= on(EjbRequestScopeActivationInterceptor.java:83) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(= ConcurrentContextInterceptor.java:45) = [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept= or.java:21) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc= essInvocation(ComponentDispatcherInterceptor.java:52) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.si = <http://org.jboss.as.ejb3.component.si/>ngleton.SingletonComponentInsta... AssociationInterceptor.processInvocation(SingletonComponentInstanceAssocia= tionInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2= 63) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j= ava:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.in = <http://org.jboss.as.ejb3.component.in/>terceptors.CurrentInvocationCon... tInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41= ) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.in = <http://org.jboss.as.ejb3.component.in/>vocationmetrics.WaitTimeInterce... r.processInvocation(WaitTimeInterceptor.java:47) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se= curityContextInterceptor.java:100) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInv= ocation(StartupAwaitInterceptor.java:22) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.in = <http://org.jboss.as.ejb3.component.in/>terceptors.ShutDownInterceptorF... ory$1.processInvocation(ShutDownInterceptorFactory.java:64) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ejb3.component.in = <http://org.jboss.as.ejb3.component.in/>terceptors.LoggingInterceptor.p... essInvocation(LoggingInterceptor.java:67) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at org.jboss.as.ee.component.Name = <http://org.jboss.as.ee.component.name/>spaceContextInterceptor.process... ocation(NamespaceContextInterceptor.java:50) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte= xtClassLoaderInterceptor.java:64) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) = at = org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur= ityManager.java:636) at = org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe= ckingInterceptor.java:61) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) = at = org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P= rivilegedWithCombinerInterceptor.java:80) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) = at = org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript= ion.java:185) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan= dler.java:73) at = org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unk= nown Source) [common.jar:] at = org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendReso= urce.java:250) at = org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen= dResource.java:182) at = org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen= dResource.java:170) at = org.ovirt.engine.api.restapi.resource.BackendHostResource.se = <http://esource.backendhostresource.se/>tupNetworks(BackendHostResource... va:212) at = org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at = org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.ja= va:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) = [rt.jar:1.8.0_111] at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 62) [rt.jar:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:= 139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMetho= dInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker= .java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour= ceLocatorInvoker.java:138) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok= er.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour= ceLocatorInvoker.java:133) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok= er.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher= .java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher= .java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.servi= ce(ServletContainerDispatcher.java:221) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht= tpServletDispatcher.java:56) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht= tpServletDispatcher.java:51) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = javax.servlet.http.HttpServlet.service(HttpServlet.java:790) = [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at = io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.j= ava:85) at = io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav= a:81) at = io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq= uest(ServletSecurityRoleHandler.java:62) at = io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl= etDispatchingHandler.java:36) at io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet= InitialHandler.java:274) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletI= nitialHandler.java:209) at = io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatch= erImpl.java:221) at = io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDis= patcherImpl.java:147) at = io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherIm= pl.java:111) at = org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt= er.java:139) at = org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt= er.java:68) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt= er.java:116) at = org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt= er.java:71) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestAp= iSessionMgmtFilter.java:78) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFi= lter.java:39) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(Sso= RestApiNegotiationFilter.java:91) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApi= AuthFilter.java:47) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SessionValidationFilter.do = <http://ers.sessionvalidationfilter.do/>Filter(SessionValidationFilter.... a:59) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(= RestApiSessionValidationFilter.java:35) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr= otectionFilter.java:111) at = org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr= otectionFilter.java:102) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSuppo= rtFilter.java:183) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav= a:84) at = io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq= uest(ServletSecurityRoleHandler.java:62) at = io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl= etDispatchingHandler.java:36) at = org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.= handleRequest(SecurityContextAssociationHandler.java:78) at = io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at = io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.han= dleRequest(SSLInformationAssociationHandler.java:131) at = io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.han= dleRequest(ServletAuthenticationCallHandler.java:57) at = io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at = io.undertow.security.handlers.AuthenticationConstraintHandler.handleReques= t(AuthenticationConstraintHandler.java:53) at = io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest= (AbstractConfidentialityHandler.java:46) at = io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHand= ler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at = io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.han= dleRequest(ServletSecurityConstraintHandler.java:59) at = io.undertow.security.handlers.AuthenticationMechanismsHandler.handleReques= t(AuthenticationMechanismsHandler.java:60) at = io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.ha= ndleRequest(CachedAuthenticatedSessionHandler.java:77) at = io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(No= tificationReceiverHandler.java:50) at = io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.ha= ndleRequest(AbstractSecurityContextAssociationHandler.java:43) at = io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at = org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRe= quest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at io.undertow.server.handlers.Pr = <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Pr... cateHandler.java:43) at = io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(Serv= letInitialHandler.java:292) at = io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletIniti= alHandler.java:81) at = io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa= ndler.java:138) at = io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa= ndler.java:135) at = io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(Ser= vletRequestContextThreadSetupAction.java:48) at = io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClass= LoaderSetupAction.java:43) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet= InitialHandler.java:272) at = io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletIniti= alHandler.java:81) at = io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(Servlet= InitialHandler.java:104) at = io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at = io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) = at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1142) [rt.jar:1.8.0_111] at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) = [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM = response: Internal timeout occured=20 right now, it might be a breaking change in VDSM. failure?=20 https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Dba= sic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l2= 91 = <https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Db= asic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l= 291> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepor= t/junit/%28root%29/004_basic_sanity/vm_run/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepo= rt/junit/%28root%29/004_basic_sanity/vm_run/> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/= exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas= ic-suite-4.1/post-004_basic_sanity.py/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact= /exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba= sic-suite-4.1/post-004_basic_sanity.py/> <https://ifireball.wordpress.com/> <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> <http://lists.ovirt.org/mailman/listinfo/devel> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/= exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas= ic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log= _ovirt-engine/engine.log = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact= /exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba= sic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_lo= g_ovirt-engine/engine.log> interesting error, Arik. Reruns should be added to lago The root cause is few lines above and the same as was fixed yesterday on = master. It just needs a backport to 4.1=20 [org.ovirt.engine.core.bll.RunVmOnceCommand] = (org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: = java.lang.IllegalStateException: Duplicate key = org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6= 0b java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) = [rt.jar:1.8.0_111] > at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111] java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) = [rt.jar:1.8.0_111] java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) = [rt.jar:1.8.0_111] java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1= 374) [rt.jar:1.8.0_111] java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) = [rt.jar:1.8.0_111] java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47= 1) [rt.jar:1.8.0_111] java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)= [rt.jar:1.8.0_111] java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) = [rt.jar:1.8.0_111] java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) = [rt.jar:1.8.0_111] org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM= ap(DiskHandler.java:46) [bll.jar:] org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar= d(RunVmValidator.java:318) [bll.jar:] org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator= .java:146) [bll.jar:] org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) = [bll.jar:] org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:= 58) [bll.jar:] org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83= 4) [bll.jar:] org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) = [bll.jar:] org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa= se.java:163) [bll.jar:] org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138= ) [bll.jar:] org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) = [bll.jar:] the issues from there. --Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 22 Dec 2016, at 08:57, Doron Fediuck &lt;<a = href=3D"mailto:dfediuck@redhat.com" class=3D&quot;&quot;&gt;dfediuck(a)redhat.com&lt;/a&gt;&amp;gt;= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Thu, Dec 22, 2016 at 9:48 AM, = Michal Skrivanek <span dir=3D"ltr" class=3D"">&lt;<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;michal.skrivanek(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D""><br class=3D""><div = class=3D""><span class=3D"gmail-"><blockquote type=3D"cite" = class=3D""><div class=3D"">On 21 Dec 2016, at 20:52, Eyal Edri &lt;<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;eedri(a)redhat.com&lt;/a&gt;&amp;gt; wrote:</div><br = class=3D"gmail-m_-6974087903737897314Apple-interchange-newline"><div = class=3D""><div dir=3D"auto" class=3D"">Not as easy at it sounds, &nbsp; = the current flow of ost is much more complicated than just build = artifacts. I'm not saving we won't do it, &nbsp;but it won't be ready = tomorrow and we designed lago exactly for being independent of any ci = system so anyone can run it on their = laptop.&nbsp;</div></div></blockquote><div class=3D""><br = class=3D""></div></span>+1</div><div class=3D"">we just need to improve = a little bit to make it possible for everyone. Almost there.</div><div = class=3D""><span class=3D"gmail-"><br class=3D""><blockquote type=3D"cite"= class=3D""><div class=3D""><div dir=3D"auto" class=3D""><div dir=3D"auto"= class=3D"">&nbsp;So maybe using Jenkins might be simpler but people = shouldn't skip verification just because such a job doesn't exist = yet.&nbsp;</div></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Dec 21, 2016 9:02 PM, "Oved Ourfali" &lt;<a = href=3D"mailto:oourfali@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;oourfali(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br type=3D"attribution" = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p = dir=3D"ltr" class=3D"">Why not run it via Jenkins for patches? <br = class=3D""> Like, if you add a comment saying "run: ost" it will run it? <br = class=3D""></p></blockquote></div></div></div></blockquote><div = class=3D""><br class=3D""></div></span>please don=E2=80=99t add more. = Even the Rerun-hooks thing is hard to remember. Even after years I = always keep asking is it "Re-run hooks=E2=80=9D or =E2=80=9CRerun-Hooks=E2= =80=9D or =E2=80=9Crerun hooks=E2=80=9D, does case = matter=E2=80=A6..bleh</div><div class=3D"">Either add the button or = maybe add it as a link in the automated comments (that would actually = work good enough)</div><div class=3D""><br class=3D""></div><div = class=3D"">Thanks,</div><div class=3D"">michal<div class=3D""><div = class=3D"gmail-h5"><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><p dir=3D"ltr" class=3D""> It do it automatically based on another thing? <br class=3D""></p><p = dir=3D"ltr" class=3D"">On Dec 21, 2016 17:42, "Eyal Edri" &lt;<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;eedri(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;<br class=3D""> &gt;<br class=3D""> &gt;<br class=3D""> &gt; On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek &lt;<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;michal.skrivanek(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt;&gt; On 21 Dec 2016, at 16:25, Yaniv Kaul &lt;<a = href=3D"mailto:ykaul@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;ykaul(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt; On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek &lt;<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;michal.skrivanek(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt; On 21 Dec 2016, at 14:56, Michal Skrivanek &lt;<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;michal.skrivanek(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; On 21 Dec 2016, at 12:19, Eyal Edri &lt;<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;eedri(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz = Feenstra &lt;<a href=3D"mailto:vfeenstr@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;vfeenstr(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; On Dec 21, 2016, at 11:17 AM, Barak = Korren &lt;<a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D&quot;&quot;&gt;bkorren(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; The test for running VMs had been = failing since yesterday.<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; The patch merged before the failures = started was:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"https://gerrit.ovirt.org/#/c/68826/" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/6<wbr class=3D"">8826/</a><br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; The error we`re seeing is a time-out = (after two minutes) while running<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; this API call:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; api.vms.get(VM0_NAME).status.s<wbr = class=3D"">tate =3D=3D =E2=80=98up'<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; This is a REST API call, the patch above is = Frontend. So this is unrelated.<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; However on Host 0 I can see this:<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) = [virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') = The vm start process failed (vm:615) Traceback (most recent call last): = File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm = self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run = self._connection.createXML(dom<wbr class=3D"">xml, flags), File = "/usr/lib/python2.7/site-packa<wbr = class=3D"">ges/vdsm/libvirtconnection.py"<wbr class=3D"">, line 123, in = wrapper ret =3D f(*args, **kwargs) File = "/usr/lib/python2.7/site-packa<wbr class=3D"">ges/vdsm/utils.py", line = 941, in wrapper return func(inst, *args, **kwargs) File = "/usr/lib64/python2.7/site-pac<wbr class=3D"">kages/libvirt.py", line = 3782, in createXML if ret is None:raise = libvirtError('virDomainCreateX<wbr class=3D"">ML() failed', conn=3Dself) = libvirtError: internal error: process exited while connecting to = monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not = present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 = 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus = should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: = -device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: = Warning: speed mismatch trying to attach usb device "QEMU USB CCID" = (full speed) to bus "usb.0", port "1" (high speed) <br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt; it is likely related to the recent USB = patches&nbsp;<br class=3D""> &gt;&gt;&gt;&gt;&gt; investigating<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt; hm, there are multiple problems (features/bugs = depending on prefferred point of view:)<br class=3D""> &gt;&gt;&gt;&gt; but there is an easy =E2=80=9Cfix=E2=80=9D taking care = of this particular problem, so we can start with that and figure out the = proper approach later<br class=3D""> &gt;&gt;&gt;&gt; arik will push that and merge it soon, likely today<br = class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt; Thanks - if there is a quicker way to resolve this by = reverting, I think it's a better option.<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt; I really need to talk you out of this approach:-)<br class=3D""> &gt;&gt; It does sound tempting and logical, but with our development = model of large patch series combined with late detection it really is = quite risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out = the right revert patch is more complicated then fixing it.<br class=3D""> &gt;<br class=3D""> &gt;<br class=3D""> &gt; Can we start asking developers run OST before they merge so it will = be early detection and not late detection?&nbsp;<br class=3D""> &gt; We have video sessions on how to use OST, so it shouldn't be any = issues in running it on a patch.<br class=3D""> &gt; &nbsp;<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt; I believe the best is to identify it early and notify the = maintainer who merged that patch ASAP, as that person is in the best = position to asses if revert is safe or if there is a simple follow up = patch he can push right away<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt; We can surely improve on reporting, so Barak, how/why did you = point to that particular patch in your email? It should start failing on = 16c2ec236184b3152f1df8e874b431<wbr class=3D"">15f78d0989 (CommitDate: = Fri Dec 16 01:56:07 2016 -0500)<br class=3D""> &gt;&gt; Even though it may be that it was hidden because = of&nbsp;c46f653a7846c3c2a76507b8dcf<wbr class=3D"">5bc0391ec5709 = (CommitDate: Mon Dec 19 15:16:40 2016 -0500)<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt; (fix is ready, waiting on CI now)<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt; Thanks,<br class=3D""> &gt;&gt; michal<br class=3D""> &gt;&gt;<br class=3D""> &gt;&gt;&gt; Y.<br class=3D""> &gt;&gt;&gt; &nbsp;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; 2016-12-20 16:54:43,550 INFO (vm/d299ab29) = [virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') = Changed state to Down: internal error: process exited while connecting = to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not = present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 = 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus = should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: = -device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: = Warning: speed mismatch trying to attach usb device "QEMU USB CCID" = (full speed) to bus "usb.0", port "1" (high speed) (code=3D1) (vm:1197) = 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] = (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') Stopping = connection (guestagent:430) <br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; And on The engine loads of these:<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; 2016-12-20 16:53:57,844-05 ERROR = [org.ovirt.engine.core.vdsbrok<wbr class=3D"">er.vdsbroker.PollVDSCommand]= (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] = Command 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, = VdsIdVDSCommandParametersBase:<wbr class=3D"">{runAsync=3D'true', = hostId=3D'994b5d79-605f-4415-94f<wbr class=3D"">2-02c79cfa246e'})' = execution failed: VDSGenericException: VDSNetworkException: Timeout = during rpc call 2016-12-20 16:53:57,849-05 DEBUG = [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.reactors.stomp.impl.Message] (SSL Stomp Reactor) [7971dfb4] = MESSAGE content-length:80 destination:jms.topic.vdsm_res<wbr = class=3D"">ponses content-type:application/json = subscription:5b6494d5-d5a0-477<wbr class=3D"">1-941c-a8be70f72450 = {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr = class=3D"">95c122d", "result": true}=EF=BF=BD 2016-12-20 16:53:57,850-05 = DEBUG [org.ovirt.vdsm.jsonrpc.client<wbr = class=3D"">.internal.ResponseWorker] (ResponseWorker) [] Message = received: {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr = class=3D"">95c122d", "result": true} 2016-12-20 16:53:57,850-05 ERROR = [org.ovirt.vdsm.jsonrpc.client<wbr class=3D"">.JsonRpcClient] = (ResponseWorker) [] Not able to update response for = "3c95fdb0-5b77-4927-9f6e-adc73<wbr class=3D"">95c122d" 2016-12-20 = 16:53:57,844-05 DEBUG [org.ovirt.engine.core.vdsbrok<wbr = class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) = [5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Exception: = org.ovirt.engine.core.vdsbroke<wbr = class=3D"">r.vdsbroker.VDSNetworkExceptio<wbr class=3D"">n: = VDSGenericException: VDSNetworkException: Timeout during rpc call = at org.ovirt.engine.core.vdsbroke<wbr = class=3D"">r.vdsbroker.FutureVDSCommand.<wbr = class=3D"">get(FutureVDSCommand.java:73) [vdsbroker.jar:] at = org.ovirt.engine.core.bll.netw<wbr = class=3D"">ork.host.HostSetupNetworkPolle<wbr = class=3D"">r.getValue(HostSetupNetworkPol<wbr class=3D"">ler.java:56) = [bll.jar:] at org.ovirt.engine.core.bll.netw<wbr = class=3D"">ork.host.HostSetupNetworkPolle<wbr = class=3D"">r.poll(HostSetupNetworkPoller.<wbr class=3D"">java:41) = [bll.jar:] at org.ovirt.engine.core.bll.netw<wbr = class=3D"">ork.host.HostSetupNetworksComm<wbr = class=3D"">and.invokeSetupNetworksCommand<wbr = class=3D"">(HostSetupNetworksCommand.<wbr class=3D"">java:426) = [bll.jar:] at org.ovirt.engine.core.bll.netw<wbr = class=3D"">ork.host.HostSetupNetworksComm<wbr = class=3D"">and.executeCommand(HostSetupNe<wbr = class=3D"">tworksCommand.java:287) [bll.jar:] at = org.ovirt.engine.core.bll.Comm<wbr = class=3D"">andBase.executeWithoutTransact<wbr = class=3D"">ion(CommandBase.java:1249) [bll.jar:] at = org.ovirt.engine.core.bll.Comm<wbr = class=3D"">andBase.executeActionInTransac<wbr = class=3D"">tionScope(CommandBase.java:<wbr class=3D"">1389) [bll.jar:] = at org.ovirt.engine.core.bll.Comm<wbr = class=3D"">andBase.runInTransaction(Comma<wbr class=3D"">ndBase.java:2053)= [bll.jar:] at <a href=3D"http://org.ovirt.engine.core.utils.tr/" = class=3D"">org.ovirt.engine.core.utils.tr</a><wbr = class=3D"">ansaction.TransactionSupport.e<wbr = class=3D"">xecuteInSuppressed(Transaction<wbr class=3D"">Support.java:164)= [utils.jar:] at <a href=3D"http://org.ovirt.engine.core.utils.tr/" = class=3D"">org.ovirt.engine.core.utils.tr</a><wbr = class=3D"">ansaction.TransactionSupport.e<wbr = class=3D"">xecuteInScope(TransactionSuppo<wbr class=3D"">rt.java:103) = [utils.jar:] at org.ovirt.engine.core.bll.Comm<wbr = class=3D"">andBase.execute(CommandBase.<wbr class=3D"">java:1449) = [bll.jar:] at org.ovirt.engine.core.bll.Comm<wbr = class=3D"">andBase.executeAction(CommandB<wbr class=3D"">ase.java:395) = [bll.jar:] at org.ovirt.engine.core.bll.exec<wbr = class=3D"">utor.DefaultBackendActionExecu<wbr = class=3D"">tor.execute(DefaultBackendActi<wbr = class=3D"">onExecutor.java:13) [bll.jar:] at = org.ovirt.engine.core.bll.Back<wbr = class=3D"">end.runAction(Backend.java:<wbr class=3D"">511) [bll.jar:] = at org.ovirt.engine.core.bll.Back<wbr = class=3D"">end.runActionImpl(Backend.java<wbr class=3D"">:493) = [bll.jar:] at org.ovirt.engine.core.bll.Back<wbr = class=3D"">end.runAction(Backend.java:<wbr class=3D"">446) [bll.jar:] = at sun.reflect.GeneratedMethodAcc<wbr class=3D"">essor232.invoke(Unknown = Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAc<wbr = class=3D"">cessorImpl.invoke(DelegatingMe<wbr = class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) = [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana<wbr = class=3D"">gedReferenceMethodInterceptor.<wbr = class=3D"">processInvocation(ManagedRefer<wbr = class=3D"">enceMethodInterceptor.java:52) at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext$Invocation.proceed(<wbr = class=3D"">InterceptorContext.java:437) at = org.jboss.as.weld.ejb.Jsr299Bi<wbr = class=3D"">ndingsInterceptor.delegateInte<wbr = class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:70) = [wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299Bi<wbr = class=3D"">ndingsInterceptor.doMethodInte<wbr = class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:80) = [wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299Bi<wbr = class=3D"">ndingsInterceptor.processInvoc<wbr = class=3D"">ation(Jsr299BindingsIntercepto<wbr class=3D"">r.java:93) = [wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.as.ee.component.inte<wbr = class=3D"">rceptors.UserInterceptorFactor<wbr = class=3D"">y$1.processInvocation(UserInte<wbr = class=3D"">rceptorFactory.java:63) at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext$Invocation.proceed(<wbr = class=3D"">InterceptorContext.java:437) at = org.ovirt.engine.core.bll.inte<wbr = class=3D"">rceptors.CorrelationIdTrackerI<wbr = class=3D"">nterceptor.aroundInvoke(Correl<wbr = class=3D"">ationIdTrackerInterceptor.<wbr class=3D"">java:13) [bll.jar:] = at sun.reflect.GeneratedMethodAcc<wbr = class=3D"">essor64.invoke(Unknown Source) [:1.8.0_111] at = sun.reflect.DelegatingMethodAc<wbr = class=3D"">cessorImpl.invoke(DelegatingMe<wbr = class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) = [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana<wbr = class=3D"">gedReferenceLifecycleMethodInt<wbr = class=3D"">erceptor.processInvocation(Man<wbr = class=3D"">agedReferenceLifecycleMethodIn<wbr = class=3D"">terceptor.java:89) at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.in/" = class=3D"">org.jboss.as.ejb3.component.in</a><wbr = class=3D"">vocationmetrics.ExecutionTimeI<wbr = class=3D"">nterceptor.processInvocation(E<wbr = class=3D"">xecutionTimeInterceptor.java:4<wbr class=3D"">3) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext$Invocation.proceed(<wbr = class=3D"">InterceptorContext.java:437) at = org.jboss.weld.ejb.AbstractEJB<wbr = class=3D"">RequestScopeActivationIntercep<wbr = class=3D"">tor.aroundInvoke(AbstractEJBRe<wbr = class=3D"">questScopeActivationIntercepto<wbr class=3D"">r.java:73) = [weld-core-impl-2.3.5.Final.ja<wbr class=3D"">r:2.3.5.Final] at = org.jboss.as.weld.ejb.EjbReque<wbr = class=3D"">stScopeActivationInterceptor.<wbr = class=3D"">processInvocation(EjbRequestSc<wbr = class=3D"">opeActivationInterceptor.java:<wbr class=3D"">83) = [wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.as.ee.concurrent.Con<wbr = class=3D"">currentContextInterceptor.proc<wbr = class=3D"">essInvocation(ConcurrentContex<wbr = class=3D"">tInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:1<wbr = class=3D"">0.1.0.Final] at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.InitialIn<wbr = class=3D"">terceptor.processInvocation(In<wbr = class=3D"">itialInterceptor.java:21) at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.ChainedIn<wbr = class=3D"">terceptor.processInvocation(Ch<wbr = class=3D"">ainedInterceptor.java:61) at = org.jboss.as.ee.component.inte<wbr = class=3D"">rceptors.ComponentDispatcherIn<wbr = class=3D"">terceptor.processInvocation(Co<wbr = class=3D"">mponentDispatcherInterceptor.<wbr class=3D"">java:52) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.si/" = class=3D"">org.jboss.as.ejb3.component.si</a><wbr = class=3D"">ngleton.SingletonComponentInst<wbr = class=3D"">anceAssociationInterceptor.pro<wbr = class=3D"">cessInvocation(SingletonCompon<wbr = class=3D"">entInstanceAssociationIntercep<wbr class=3D"">tor.java:53) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.as.ejb3.tx.CMTTxInte<wbr = class=3D"">rceptor.invokeInNoTx(CMTTxInte<wbr class=3D"">rceptor.java:263)= [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInte<wbr = class=3D"">rceptor.supports(CMTTxIntercep<wbr class=3D"">tor.java:374) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInte<wbr = class=3D"">rceptor.processInvocation(CMTT<wbr = class=3D"">xInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar<wbr = class=3D"">:10.1.0.Final] at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.in/" = class=3D"">org.jboss.as.ejb3.component.in</a><wbr = class=3D"">terceptors.CurrentInvocationCo<wbr = class=3D"">ntextInterceptor.<wbr = class=3D"">processInvocation(CurrentInvoc<wbr = class=3D"">ationContextInterceptor.java:<wbr class=3D"">41) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.in/" = class=3D"">org.jboss.as.ejb3.component.in</a><wbr = class=3D"">vocationmetrics.WaitTimeInterc<wbr = class=3D"">eptor.processInvocation(WaitTi<wbr = class=3D"">meInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar<wbr = class=3D"">:10.1.0.Final] at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.as.ejb3.security.Sec<wbr = class=3D"">urityContextInterceptor.proces<wbr = class=3D"">sInvocation(SecurityContextInt<wbr = class=3D"">erceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar<wbr = class=3D"">:10.1.0.Final] at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.as.ejb3.deployment.p<wbr = class=3D"">rocessors.StartupAwaitIntercep<wbr = class=3D"">tor.processInvocation(StartupA<wbr = class=3D"">waitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar<wbr = class=3D"">:10.1.0.Final] at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.in/" = class=3D"">org.jboss.as.ejb3.component.in</a><wbr = class=3D"">terceptors.ShutDownInterceptor<wbr = class=3D"">Factory$1.processInvocation(Sh<wbr = class=3D"">utDownInterceptorFactory.java:<wbr class=3D"">64) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ejb3.component.in/" = class=3D"">org.jboss.as.ejb3.component.in</a><wbr = class=3D"">terceptors.LoggingInterceptor.<wbr = class=3D"">processInvocation(LoggingInter<wbr class=3D"">ceptor.java:67) = [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at <a href=3D"http://org.jboss.as.ee.component.name/" = class=3D"">org.jboss.as.ee.component.Name</a><wbr = class=3D"">spaceContextInterceptor.proces<wbr = class=3D"">sInvocation(NamespaceContextIn<wbr = class=3D"">terceptor.java:50) at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.ContextCl<wbr = class=3D"">assLoaderInterceptor.processIn<wbr = class=3D"">vocation(ContextClassLoaderInt<wbr class=3D"">erceptor.java:64)= at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) = at org.wildfly.security.manager.W<wbr = class=3D"">ildFlySecurityManager.doChecke<wbr = class=3D"">d(WildFlySecurityManager.java:<wbr class=3D"">636) at = org.jboss.invocation.AccessChe<wbr = class=3D"">ckingInterceptor.processInvoca<wbr = class=3D"">tion(AccessCheckingInterceptor<wbr class=3D"">.java:61) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.Intercept<wbr = class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) = at org.jboss.invocation.Privilege<wbr = class=3D"">dWithCombinerInterceptor.<wbr = class=3D"">processInvocation(PrivilegedWi<wbr = class=3D"">thCombinerInterceptor.java:80) at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.ChainedIn<wbr = class=3D"">terceptor.processInvocation(Ch<wbr = class=3D"">ainedInterceptor.java:61) at = org.jboss.as.ee.component.View<wbr = class=3D"">Service$View.invoke(ViewServic<wbr class=3D"">e.java:198) = at org.jboss.as.ee.component.View<wbr = class=3D"">Description$1.processInvocatio<wbr = class=3D"">n(ViewDescription.java:185) at = org.jboss.invocation.Intercept<wbr = class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) = at org.jboss.invocation.ChainedIn<wbr = class=3D"">terceptor.processInvocation(Ch<wbr = class=3D"">ainedInterceptor.java:61) at = org.jboss.as.ee.component.Prox<wbr = class=3D"">yInvocationHandler.invoke(Prox<wbr = class=3D"">yInvocationHandler.java:73) at = org.ovirt.engine.core.common.i<wbr = class=3D"">nterfaces.BackendLocal$$$view4<wbr = class=3D"">.runAction(Unknown Source) [common.jar:] at = org.ovirt.engine.api.restapi.r<wbr = class=3D"">esource.BackendResource.doActi<wbr = class=3D"">on(BackendResource.java:250) at = org.ovirt.engine.api.restapi.r<wbr = class=3D"">esource.BackendResource.perfor<wbr = class=3D"">mAction(BackendResource.java:<wbr class=3D"">182) at = org.ovirt.engine.api.restapi.r<wbr = class=3D"">esource.BackendResource.perfor<wbr = class=3D"">mAction(BackendResource.java:<wbr class=3D"">170) at = org.ovirt.engine.api.restapi.r<wbr class=3D""><a = href=3D"http://esource.backendhostresource.se/" = class=3D"">esource.BackendHostResource.se</a><wbr = class=3D"">tupNetworks(BackendHostResourc<wbr class=3D"">e.java:212) = at org.ovirt.engine.api.v3.V3Serv<wbr = class=3D"">er.adaptAction(V3Server.java:<wbr class=3D"">216) at = org.ovirt.engine.api.v3.server<wbr = class=3D"">s.V3HostServer.setupNetworks(<wbr = class=3D"">V3HostServer.java:182) at = sun.reflect.NativeMethodAccess<wbr class=3D"">orImpl.invoke0(Native = Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccess<wbr = class=3D"">orImpl.invoke(NativeMethodAcce<wbr class=3D"">ssorImpl.java:62)= [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAc<wbr = class=3D"">cessorImpl.invoke(DelegatingMe<wbr = class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) = [rt.jar:1.8.0_111] at org.jboss.resteasy.core.Method<wbr = class=3D"">InjectorImpl.invoke(MethodInje<wbr = class=3D"">ctorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.core.Resour<wbr = class=3D"">ceMethodInvoker.invokeOnTarget<wbr = class=3D"">(ResourceMethodInvoker.java:29<wbr class=3D"">5) = [resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] at = org.jboss.resteasy.core.Resour<wbr = class=3D"">ceMethodInvoker.invoke(Resourc<wbr = class=3D"">eMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.core.Resour<wbr = class=3D"">ceLocatorInvoker.invokeOnTarge<wbr = class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:138) = [resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] at = org.jboss.resteasy.core.Resour<wbr = class=3D"">ceLocatorInvoker.invoke(Resour<wbr = class=3D"">ceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.core.Resour<wbr = class=3D"">ceLocatorInvoker.invokeOnTarge<wbr = class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:133) = [resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] at = org.jboss.resteasy.core.Resour<wbr = class=3D"">ceLocatorInvoker.invoke(Resour<wbr = class=3D"">ceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.core.Synchr<wbr = class=3D"">onousDispatcher.invoke(Synchro<wbr = class=3D"">nousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.core.Synchr<wbr = class=3D"">onousDispatcher.invoke(Synchro<wbr = class=3D"">nousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.plugins.ser<wbr = class=3D"">ver.servlet.ServletContainerDi<wbr = class=3D"">spatcher.service(ServletContai<wbr = class=3D"">nerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.j<wbr = class=3D"">ar:3.0.19.Final] at org.jboss.resteasy.plugins.ser<wbr = class=3D"">ver.servlet.HttpServletDispatc<wbr = class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:56) = [resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] at = org.jboss.resteasy.plugins.ser<wbr = class=3D"">ver.servlet.HttpServletDispatc<wbr = class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:51) = [resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] at = javax.servlet.http.HttpServlet<wbr = class=3D"">.service(HttpServlet.java:790) = [jboss-servlet-api_3.1_spec-1.<wbr class=3D"">0.0.Final.jar:1.0.0.Final] = at io.undertow.servlet.handlers.S<wbr = class=3D"">ervletHandler.handleRequest(Se<wbr = class=3D"">rvletHandler.java:85) at = io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler.handleRequest(Fil<wbr = class=3D"">terHandler.java:81) at io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.ServletSecurityRoleHan<wbr = class=3D"">dler.handleRequest(ServletSecu<wbr = class=3D"">rityRoleHandler.java:62) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletDispatchingHandler.handl<wbr = class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) = at <a href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at <a = href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at <a = href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.dispatchR<wbr = class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:274) = at io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.dispatchT<wbr = class=3D"">oPath(ServletInitialHandler.<wbr class=3D"">java:209) = at io.undertow.servlet.spec.Reque<wbr = class=3D"">stDispatcherImpl.forwardImpl(R<wbr = class=3D"">equestDispatcherImpl.java:221) at = io.undertow.servlet.spec.Reque<wbr = class=3D"">stDispatcherImpl.forwardImplSe<wbr = class=3D"">tup(RequestDispatcherImpl.<wbr class=3D"">java:147) at = io.undertow.servlet.spec.Reque<wbr = class=3D"">stDispatcherImpl.forward(Reque<wbr = class=3D"">stDispatcherImpl.java:111) at = org.ovirt.engine.api.restapi.i<wbr = class=3D"">nvocation.VersionFilter.doFilt<wbr = class=3D"">er(VersionFilter.java:139) at = org.ovirt.engine.api.restapi.i<wbr = class=3D"">nvocation.VersionFilter.doFilt<wbr = class=3D"">er(VersionFilter.java:68) at = io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.api.restapi.i<wbr = class=3D"">nvocation.CurrentFilter.doFilt<wbr = class=3D"">er(CurrentFilter.java:116) at = org.ovirt.engine.api.restapi.i<wbr = class=3D"">nvocation.CurrentFilter.doFilt<wbr = class=3D"">er(CurrentFilter.java:71) at = io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr = class=3D"">ers.RestApiSessionMgmtFilter.d<wbr = class=3D"">oFilter(RestApiSessionMgmtFilt<wbr class=3D"">er.java:78) = [aaa.jar:] at io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr = class=3D"">ers.EnforceAuthFilter.doFilter<wbr = class=3D"">(EnforceAuthFilter.java:39) [aaa.jar:] at = io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr = class=3D"">ers.SsoRestApiNegotiationFilte<wbr = class=3D"">r.doFilter(SsoRestApiNegotiati<wbr class=3D"">onFilter.java:91)= [aaa.jar:] at io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr = class=3D"">ers.SsoRestApiAuthFilter.doFil<wbr = class=3D"">ter(SsoRestApiAuthFilter.java:<wbr class=3D"">47) [aaa.jar:] = at io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr class=3D""><a = href=3D"http://ers.sessionvalidationfilter.do/" = class=3D"">ers.SessionValidationFilter.do</a><wbr = class=3D"">Filter(SessionValidationFilter<wbr class=3D"">.java:59) = [aaa.jar:] at io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.core.aaa.filt<wbr = class=3D"">ers.RestApiSessionValidationFi<wbr = class=3D"">lter.doFilter(RestApiSessionVa<wbr = class=3D"">lidationFilter.java:35) [aaa.jar:] at = io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.api.restapi.s<wbr = class=3D"">ecurity.CSRFProtectionFilter.d<wbr = class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:111) = at org.ovirt.engine.api.restapi.s<wbr = class=3D"">ecurity.CSRFProtectionFilter.d<wbr = class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:102) = at io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = org.ovirt.engine.api.restapi.s<wbr = class=3D"">ecurity.CORSSupportFilter.doFi<wbr = class=3D"">lter(CORSSupportFilter.java:<wbr class=3D"">183) at = io.undertow.servlet.core.Manag<wbr = class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) = at io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler$FilterChainImpl.d<wbr = class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) at = io.undertow.servlet.handlers.F<wbr = class=3D"">ilterHandler.handleRequest(Fil<wbr = class=3D"">terHandler.java:84) at io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.ServletSecurityRoleHan<wbr = class=3D"">dler.handleRequest(ServletSecu<wbr = class=3D"">rityRoleHandler.java:62) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletDispatchingHandler.handl<wbr = class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) = at org.wildfly.extension.undertow<wbr = class=3D"">.security.SecurityContextAssoc<wbr = class=3D"">iationHandler.handleRequest(Se<wbr = class=3D"">curityContextAssociationHandle<wbr class=3D"">r.java:78) = at <a href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at = io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.SSLInformationAssociat<wbr = class=3D"">ionHandler.handleRequest(SSLIn<wbr = class=3D"">formationAssociationHandler.<wbr class=3D"">java:131) = at io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.ServletAuthenticationC<wbr = class=3D"">allHandler.handleRequest(Servl<wbr = class=3D"">etAuthenticationCallHandler.<wbr class=3D"">java:57) = at <a href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at = io.undertow.security.handlers.<wbr = class=3D"">AuthenticationConstraintHandle<wbr = class=3D"">r.handleRequest(Authentication<wbr = class=3D"">ConstraintHandler.java:53) at = io.undertow.security.handlers.<wbr = class=3D"">AbstractConfidentialityHandler<wbr = class=3D"">.handleRequest(AbstractConfide<wbr = class=3D"">ntialityHandler.java:46) at = io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.ServletConfidentiality<wbr = class=3D"">ConstraintHandler.<wbr = class=3D"">handleRequest(ServletConfident<wbr = class=3D"">ialityConstraintHandler.java:<wbr class=3D"">64) at = io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.ServletSecurityConstra<wbr = class=3D"">intHandler.handleRequest(Servl<wbr = class=3D"">etSecurityConstraintHandler.<wbr class=3D"">java:59) = at io.undertow.security.handlers.<wbr = class=3D"">AuthenticationMechanismsHandle<wbr = class=3D"">r.handleRequest(Authentication<wbr = class=3D"">MechanismsHandler.java:60) at = io.undertow.servlet.handlers.s<wbr = class=3D"">ecurity.CachedAuthenticatedSes<wbr = class=3D"">sionHandler.handleRequest(Cach<wbr = class=3D"">edAuthenticatedSessionHandler.<wbr class=3D"">java:77) = at io.undertow.security.handlers.<wbr = class=3D"">NotificationReceiverHandler.ha<wbr = class=3D"">ndleRequest(NotificationReceiv<wbr = class=3D"">erHandler.java:50) at io.undertow.security.handlers.<wbr = class=3D"">AbstractSecurityContextAssocia<wbr = class=3D"">tionHandler.handleRequest(Abst<wbr = class=3D"">ractSecurityContextAssociation<wbr class=3D"">Handler.java:43) = at <a href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at = org.wildfly.extension.undertow<wbr = class=3D"">.security.jacc.JACCContextIdHa<wbr = class=3D"">ndler.handleRequest(JACCContex<wbr = class=3D"">tIdHandler.java:61) at <a = href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at <a = href=3D"http://io.undertow.server.handlers.pr/" = class=3D"">io.undertow.server.handlers.Pr</a><wbr = class=3D"">edicateHandler.handleRequest(P<wbr = class=3D"">redicateHandler.java:43) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.handleFir<wbr = class=3D"">stRequest(ServletInitialHandle<wbr class=3D"">r.java:292) = at io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.access$10<wbr = class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler$2.call(Se<wbr = class=3D"">rvletInitialHandler.java:138) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler$2.call(Se<wbr = class=3D"">rvletInitialHandler.java:135) at = io.undertow.servlet.core.Servl<wbr = class=3D"">etRequestContextThreadSetupAct<wbr = class=3D"">ion$1.call(ServletRequestConte<wbr = class=3D"">xtThreadSetupAction.java:48) at = io.undertow.servlet.core.Conte<wbr = class=3D"">xtClassLoaderSetupAction$1.<wbr = class=3D"">call(ContextClassLoaderSetupAc<wbr class=3D"">tion.java:43) = at io.undertow.servlet.api.Legacy<wbr = class=3D"">ThreadSetupActionWrapper$1.<wbr = class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) = at io.undertow.servlet.api.Legacy<wbr = class=3D"">ThreadSetupActionWrapper$1.<wbr = class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) = at io.undertow.servlet.api.Legacy<wbr = class=3D"">ThreadSetupActionWrapper$1.<wbr = class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) = at io.undertow.servlet.api.Legacy<wbr = class=3D"">ThreadSetupActionWrapper$1.<wbr = class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) = at io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.dispatchR<wbr = class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:272) = at io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler.access$00<wbr = class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) at = io.undertow.servlet.handlers.S<wbr = class=3D"">ervletInitialHandler$1.handleR<wbr = class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:104) = at io.undertow.server.Connectors.<wbr = class=3D"">executeRootHandler(Connectors.<wbr class=3D"">java:202) = at io.undertow.server.HttpServerE<wbr = class=3D"">xchange$1.run(HttpServerExchan<wbr class=3D"">ge.java:805) = at java.util.concurrent.ThreadPoo<wbr = class=3D"">lExecutor.runWorker(ThreadPool<wbr = class=3D"">Executor.java:1142) [rt.jar:1.8.0_111] at = java.util.concurrent.ThreadPoo<wbr = class=3D"">lExecutor$Worker.run(ThreadPoo<wbr = class=3D"">lExecutor.java:617) [rt.jar:1.8.0_111] at = java.lang.Thread.run(Thread.ja<wbr class=3D"">va:745) [rt.jar:1.8.0_111] = 2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbrok<wbr = class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) = [5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Timeout waiting = for VDSM response: Internal timeout occured <br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; This is definitely not related to the patch = linked above.<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; However I am also not quite sure what is = the cause of this right now, it might be a breaking change in VDSM.<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; The DC level is 4.0, is it possible that is the = reason for the failure?&nbsp;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; &nbsp;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Full test code can be seen here:<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a... b;f=3Dbasic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/m= aster#l291" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/gitwe<wbr = class=3D"">b?p=3Dovirt-system-tests.git;a=3D<wbr = class=3D"">blob;f=3Dbasic-suite-4.1/test-<wbr = class=3D"">scenarios/004_basic_sanity.py;<wbr = class=3D"">hb=3Drefs/heads/master#l291</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Full test exception can be seen = here:<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.... estReport/junit/%28root%29/004_basic_sanity/vm_run/" target=3D"_blank" = class=3D"">http://jenkins.ovirt.org/job/t<wbr = class=3D"">est-repo_ovirt_experimental_4.<wbr = class=3D"">1/13/testReport/junit/%28root%<wbr = class=3D"">29/004_basic_sanity/vm_run/</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Further logs can be seen in Jenkins:<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.... rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_= logs/basic-suite-4.1/post-004_basic_sanity.py/" target=3D"_blank" = class=3D"">http://jenkins.ovirt.org/job/t<wbr = class=3D"">est-repo_ovirt_experimental_4.<wbr = class=3D"">1/13/artifact/exported-artifac<wbr = class=3D"">ts/basic_suite_4.1.sh-el7/<wbr = class=3D"">exported-artifacts/test_logs/<wbr = class=3D"">basic-suite-4.1/post-004_basic<wbr = class=3D"">_sanity.py/</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; -- <br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Barak Korren<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a href=3D"mailto:bkorren@redhat.com" = target=3D"_blank" class=3D&quot;&quot;&gt;bkorren(a)redhat.com&lt;/a&gt;&lt;br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; RHCE, RHCi, RHV-DevOps Team<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"https://ifireball.wordpress.com/" target=3D"_blank" = class=3D"">https://ifireball.wordpress.co<wbr class=3D"">m/</a><br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; ______________________________<wbr = class=3D"">_________________<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Devel mailing list<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; ______________________________<wbr = class=3D"">_________________<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; Devel mailing list<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; -- <br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; Eyal Edri<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; Associate Manager<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; RHV DevOps<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; EMEA ENG Virtualization R&amp;D<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; Red Hat Israel<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; phone: <a href=3D"tel:+972%209-769-2018" = value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; irc: eedri (on #tlv #rhev-dev #rhev-integ)<br = class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; ______________________________<wbr = class=3D"">_________________<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; Devel mailing list<br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> &gt;&gt;&gt;&gt;&gt;&gt; <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;&gt; ______________________________<wbr = class=3D"">_________________<br class=3D""> &gt;&gt;&gt;&gt; Devel mailing list<br class=3D""> &gt;&gt;&gt;&gt; <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> &gt;&gt;&gt;&gt; <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel"= target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;&gt;<br class=3D""> &gt;&gt;<br class=3D""> &gt;<br class=3D""> &gt;<br class=3D""> &gt;<br class=3D""> &gt; -- <br class=3D""> &gt; Eyal Edri<br class=3D""> &gt; Associate Manager<br class=3D""> &gt; RHV DevOps<br class=3D""> &gt; EMEA ENG Virtualization R&amp;D<br class=3D""> &gt; Red Hat Israel<br class=3D""> &gt;<br class=3D""> &gt; phone: <a href=3D"tel:+972%209-769-2018" value=3D"+97297692018" = target=3D"_blank" class=3D"">+972-9-7692018</a><br class=3D""> &gt; irc: eedri (on #tlv #rhev-dev #rhev-integ)<br class=3D""> &gt;<br class=3D""> &gt; ______________________________<wbr class=3D"">_________________<br = class=3D""> &gt; Devel mailing list<br class=3D""> &gt; <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br class=3D""> &gt; <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""></p> </blockquote></div></div> </div></blockquote></div></div></div><br class=3D""></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" class=3D&quot;&quot;&gt;Devel(a)ovirt.org&lt;/a&gt;&lt;br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br= class=3D""></div><div class=3D"gmail_extra">Guys,</div><div = class=3D"gmail_extra">4.1 is broken for 11 hours now.</div><div = class=3D"gmail_extra">It seems that the issues are accumulating; looking = into&nbsp;<a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.... rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_= logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/= _var_log_ovirt-engine/engine.log" = class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_expe... 8/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/te= st_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engi= ne/_var_log_ovirt-engine/engine.log</a></div><div = class=3D"gmail_extra"><br class=3D""></div><div = class=3D"gmail_extra">shows a duplicate entry in disk = mapping:</div></div></div></blockquote><div><br = class=3D""></div>interesting error, Arik. Reruns should be added to = lago</div><div>The root cause is few lines above and the same as was = fixed yesterday on master. It just needs a backport to 4.1&nbsp;<br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><pre style=3D"" = class=3D"">2016-12-21 10:52:51,436-05 ERROR = [org.ovirt.engine.core.bll.RunVmOnceCommand] = (org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: = java.lang.IllegalStateException: Duplicate key = org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6= 0b at = java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) = [rt.jar:1.8.0_111] at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111] at = java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) = [rt.jar:1.8.0_111] at = java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) = [rt.jar:1.8.0_111] at = java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1= 374) [rt.jar:1.8.0_111] at = java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) = [rt.jar:1.8.0_111] at = java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47= 1) [rt.jar:1.8.0_111] at = java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)= [rt.jar:1.8.0_111] at = java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) = [rt.jar:1.8.0_111] at = java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) = [rt.jar:1.8.0_111] at = org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM= ap(DiskHandler.java:46) [bll.jar:] at = org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar= d(RunVmValidator.java:318) [bll.jar:] at = org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator= .java:146) [bll.jar:] at = org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) = [bll.jar:] at = org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:= 58) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83= 4) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) = [bll.jar:] at = org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa= se.java:163) [bll.jar:] at = org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138= ) [bll.jar:] at = org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) = [bll.jar:]</pre><pre style=3D"" class=3D""><br class=3D""></pre><pre = style=3D"" class=3D"">I suggest we consider reverting to the last good = state, and work out the issues from there.</pre></div></div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3--