[ovirt-devel] System tests for 4.1 currently failing to run VMs!

Doron Fediuck dfediuck at redhat.com
Thu Dec 22 07:57:50 UTC 2016 

On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek <
michal.skrivanek at redhat.com> wrote:

>
> On 21 Dec 2016, at 20:52, Eyal Edri <eedri at redhat.com> wrote:
>
> Not as easy at it sounds,   the current flow of ost is much more
> complicated than just build artifacts. I'm not saving we won't do it,  but
> it won't be ready tomorrow and we designed lago exactly for being
> independent of any ci system so anyone can run it on their laptop.
>
>
> +1
> we just need to improve a little bit to make it possible for everyone.
> Almost there.
>
>  So maybe using Jenkins might be simpler but people shouldn't skip
> verification just because such a job doesn't exist yet.
>
> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <oourfali at redhat.com> wrote:
>
>> Why not run it via Jenkins for patches?
>> Like, if you add a comment saying "run: ost" it will run it?
>>
>
> please don’t add more. Even the Rerun-hooks thing is hard to remember.
> Even after years I always keep asking is it "Re-run hooks” or “Rerun-Hooks”
> or “rerun hooks”, does case matter…..bleh
> Either add the button or maybe add it as a link in the automated comments
> (that would actually work good enough)
>
> Thanks,
> michal
>
> It do it automatically based on another thing?
>>
>> On Dec 21, 2016 17:42, "Eyal Edri" <eedri at redhat.com> wrote:
>> >
>> >
>> >
>> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek <
>> michal.skrivanek at redhat.com> wrote:
>> >>
>> >>
>> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul at redhat.com> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek <
>> michal.skrivanek at redhat.com> wrote:
>> >>>>
>> >>>>
>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek <
>> michal.skrivanek at redhat.com> wrote:
>> >>>>>
>> >>>>>
>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <eedri at redhat.com> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra <
>> vfeenstr at redhat.com> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <bkorren at redhat.com>
>> wrote:
>> >>>>>>>>
>> >>>>>>>> The test for running VMs had been failing since yesterday.
>> >>>>>>>>
>> >>>>>>>> The patch merged before the failures started was:
>> >>>>>>>> https://gerrit.ovirt.org/#/c/68826/
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) while
>> running
>> >>>>>>>> this API call:
>> >>>>>>>>
>> >>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up'
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is a REST API call, the patch above is Frontend. So this is
>> unrelated.
>> >>>>>>>
>> >>>>>>> However on Host 0 I can see this:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm]
>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process
>> failed (vm:615) Traceback (most recent call last): File
>> "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm self._run()
>> File "/usr/share/vdsm/virt/vm.py", line 1991, in _run
>> self._connection.createXML(domxml, flags), File
>> "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123,
>> in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py",
>> line 941, in wrapper return func(inst, *args, **kwargs) File
>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML
>> if ret is None:raise libvirtError('virDomainCreateXML() failed',
>> conn=self) libvirtError: internal error: process exited while connecting to
>> monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present
>> in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
>> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus
>> should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm:
>> -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch
>> trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0",
>> port "1" (high speed)
>> >>>>>
>> >>>>>
>> >>>>> it is likely related to the recent USB patches
>> >>>>> investigating
>> >>>>
>> >>>>
>> >>>> hm, there are multiple problems (features/bugs depending on
>> prefferred point of view:)
>> >>>> but there is an easy “fix” taking care of this particular problem,
>> so we can start with that and figure out the proper approach later
>> >>>> arik will push that and merge it soon, likely today
>> >>>
>> >>>
>> >>> Thanks - if there is a quicker way to resolve this by reverting, I
>> think it's a better option.
>> >>
>> >>
>> >> I really need to talk you out of this approach:-)
>> >> It does sound tempting and logical, but with our development model of
>> large patch series combined with late detection it really is quite risky.
>> Here it wouldn’t help much…and figuring out the right revert patch is more
>> complicated then fixing it.
>> >
>> >
>> > Can we start asking developers run OST before they merge so it will be
>> early detection and not late detection?
>> > We have video sessions on how to use OST, so it shouldn't be any issues
>> in running it on a patch.
>> >
>> >>
>> >> I believe the best is to identify it early and notify the maintainer
>> who merged that patch ASAP, as that person is in the best position to asses
>> if revert is safe or if there is a simple follow up patch he can push right
>> away
>> >>
>> >> We can surely improve on reporting, so Barak, how/why did you point to
>> that particular patch in your email? It should start failing on
>> 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16
>> 01:56:07 2016 -0500)
>> >> Even though it may be that it was hidden because
>> of c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19
>> 15:16:40 2016 -0500)
>> >>
>> >> (fix is ready, waiting on CI now)
>> >>
>> >> Thanks,
>> >> michal
>> >>
>> >>> Y.
>> >>>
>> >>>>
>> >>>>
>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm]
>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down:
>> internal error: process exited while connecting to monitor:
>> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any
>> NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z
>> qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA
>> config 2016-12-20T21:54:43.101886Z qemu-kvm: -device
>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to
>> attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1"
>> (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 INFO (vm/d299ab29)
>> [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping
>> connection (guestagent:430)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> And on The engine loads of these:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command
>> 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0,
>> VdsIdVDSCommandParametersBase:{runAsync='true',
>> hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed:
>> VDSGenericException: VDSNetworkException: Timeout during rpc call
>> 2016-12-20 16:53:57,849-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
>> (SSL Stomp Reactor) [7971dfb4] MESSAGE content-length:80
>> destination:jms.topic.vdsm_responses content-type:application/json
>> subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0",
>> "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}�
>> 2016-12-20 16:53:57,850-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
>> (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id":
>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20
>> 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient]
>> (ResponseWorker) [] Not able to update response for
>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>> task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception:
>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>> VDSGenericException: VDSNetworkException: Timeout during rpc call at
>> org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73)
>> [vdsbroker.jar:] at org.ovirt.engine.core.bll.netw
>> ork.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56)
>> [bll.jar:] at org.ovirt.engine.core.bll.netw
>> ork.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41)
>> [bll.jar:] at org.ovirt.engine.core.bll.netw
>> ork.host.HostSetupNetworksCommand.invokeSetupNetworksCommand
>> (HostSetupNetworksCommand.java:426) [bll.jar:] at
>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksComm
>> and.executeCommand(HostSetupNetworksCommand.java:287) [bll.jar:] at
>> org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249)
>> [bll.jar:] at org.ovirt.engine.core.bll.Comm
>> andBase.executeActionInTransactionScope(CommandBase.java:1389)
>> [bll.jar:] at org.ovirt.engine.core.bll.Comm
>> andBase.runInTransaction(CommandBase.java:2053) [bll.jar:] at
>> org.ovirt.engine.core.utils.transaction.TransactionSupport.e
>> xecuteInSuppressed(TransactionSupport.java:164) [utils.jar:] at
>> org.ovirt.engine.core.utils.transaction.TransactionSupport.e
>> xecuteInScope(TransactionSupport.java:103) [utils.jar:] at
>> org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449)
>> [bll.jar:] at org.ovirt.engine.core.bll.Comm
>> andBase.executeAction(CommandBase.java:395) [bll.jar:] at
>> org.ovirt.engine.core.bll.executor.DefaultBackendActionExecu
>> tor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at
>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:]
>> at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493)
>> [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446)
>> [bll.jar:] at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown
>> Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAc
>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498)
>> [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana
>> gedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
>> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
>> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInte
>> rception(Jsr299BindingsInterceptor.java:70)
>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInte
>> rception(Jsr299BindingsInterceptor.java:80)
>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvoc
>> ation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
>> at org.jboss.as.ee.component.interceptors.UserInterceptorFactor
>> y$1.processInvocation(UserInterceptorFactory.java:63) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
>> at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerI
>> nterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13)
>> [bll.jar:] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown
>> Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAc
>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498)
>> [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana
>> gedReferenceLifecycleMethodInterceptor.processInvocation(Man
>> agedReferenceLifecycleMethodInterceptor.java:89) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeI
>> nterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
>> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationIntercep
>> tor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
>> [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at
>> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.
>> processInvocation(EjbRequestScopeActivationInterceptor.java:83)
>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.proc
>> essInvocation(ConcurrentContextInterceptor.java:45)
>> [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
>> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
>> at org.jboss.as.ee.component.interceptors.ComponentDispatcherIn
>> terceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.singleton.SingletonComponentInst
>> anceAssociationInterceptor.processInvocation(SingletonCompon
>> entInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationCo
>> ntextInterceptor.processInvocation(CurrentInvoc
>> ationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
>> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterc
>> eptor.processInvocation(WaitTimeInterceptor.java:47)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.security.SecurityContextInterceptor.proces
>> sInvocation(SecurityContextInterceptor.java:100)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.deployment.processors.StartupAwaitIntercep
>> tor.processInvocation(StartupAwaitInterceptor.java:22)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptor
>> Factory$1.processInvocation(ShutDownInterceptorFactory.java:64)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.
>> processInvocation(LoggingInterceptor.java:67)
>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.as.ee.component.NamespaceContextInterceptor.proces
>> sInvocation(NamespaceContextInterceptor.java:50) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.ContextClassLoaderInterceptor.processIn
>> vocation(ContextClassLoaderInterceptor.java:64) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
>> at org.wildfly.security.manager.WildFlySecurityManager.doChecke
>> d(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessChe
>> ckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
>> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.
>> processInvocation(PrivilegedWithCombinerInterceptor.java:80) at
>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
>> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
>> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
>> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
>> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
>> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
>> at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown
>> Source) [common.jar:] at org.ovirt.engine.api.restapi.r
>> esource.BackendResource.doAction(BackendResource.java:250) at
>> org.ovirt.engine.api.restapi.resource.BackendResource.perfor
>> mAction(BackendResource.java:182) at org.ovirt.engine.api.restapi.r
>> esource.BackendResource.performAction(BackendResource.java:170) at
>> org.ovirt.engine.api.restapi.resource.BackendHostResource.se
>> tupNetworks(BackendHostResource.java:212) at
>> org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at
>> org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccess
>> orImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498)
>> [rt.jar:1.8.0_111] at org.jboss.resteasy.core.Method
>> InjectorImpl.invoke(MethodInjectorImpl.java:139)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget
>> (ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]
>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge
>> tObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]
>> at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge
>> tObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]
>> at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDi
>> spatcher.service(ServletContainerDispatcher.java:221)
>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at
>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc
>> her.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]
>> at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc
>> her.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at
>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)
>> at io.undertow.servlet.handlers.security.ServletSecurityRoleHan
>> dler.handleRequest(ServletSecurityRoleHandler.java:62) at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handl
>> eRequest(ServletDispatchingHandler.java:36) at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchR
>> equest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.S
>> ervletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at
>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)
>> at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSe
>> tup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.Reque
>> stDispatcherImpl.forward(RequestDispatcherImpl.java:111) at
>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139)
>> at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68)
>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.i
>> nvocation.CurrentFilter.doFilter(CurrentFilter.java:116) at
>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71)
>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78)
>> [aaa.jar:] at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91)
>> [aaa.jar:] at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47)
>> [aaa.jar:] at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.SessionValidationFilter.doFilter(SessionValidationFilter.java:59)
>> [aaa.jar:] at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt
>> ers.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35)
>> [aaa.jar:] at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.s
>> ecurity.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at
>> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.d
>> oFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.Manag
>> edFilter.doFilter(ManagedFilter.java:61) at
>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.s
>> ecurity.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at
>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d
>> oFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.F
>> ilterHandler.handleRequest(FilterHandler.java:84) at
>> io.undertow.servlet.handlers.security.ServletSecurityRoleHan
>> dler.handleRequest(ServletSecurityRoleHandler.java:62) at
>> io.undertow.servlet.handlers.ServletDispatchingHandler.handl
>> eRequest(ServletDispatchingHandler.java:36) at
>> org.wildfly.extension.undertow.security.SecurityContextAssoc
>> iationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.servlet.handlers.security.SSLInformationAssociat
>> ionHandler.handleRequest(SSLInformationAssociationHandler.java:131) at
>> io.undertow.servlet.handlers.security.ServletAuthenticationC
>> allHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.security.handlers.AuthenticationConstraintHandle
>> r.handleRequest(AuthenticationConstraintHandler.java:53) at
>> io.undertow.security.handlers.AbstractConfidentialityHandler
>> .handleRequest(AbstractConfidentialityHandler.java:46) at
>> io.undertow.servlet.handlers.security.ServletConfidentiality
>> ConstraintHandler.handleRequest(ServletConfident
>> ialityConstraintHandler.java:64) at io.undertow.servlet.handlers.s
>> ecurity.ServletSecurityConstraintHandler.handleRequest(Servl
>> etSecurityConstraintHandler.java:59) at io.undertow.security.handlers.
>> AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>> at io.undertow.servlet.handlers.security.CachedAuthenticatedSes
>> sionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at
>> io.undertow.security.handlers.NotificationReceiverHandler.ha
>> ndleRequest(NotificationReceiverHandler.java:50) at
>> io.undertow.security.handlers.AbstractSecurityContextAssocia
>> tionHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHa
>> ndler.handleRequest(JACCContextIdHandler.java:61) at
>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>> at io.undertow.servlet.handlers.ServletInitialHandler.handleFir
>> stRequest(ServletInitialHandler.java:292) at
>> io.undertow.servlet.handlers.ServletInitialHandler.access$10
>> 0(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.S
>> ervletInitialHandler$2.call(ServletInitialHandler.java:138) at
>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
>> at io.undertow.servlet.core.ServletRequestContextThreadSetupAct
>> ion$1.call(ServletRequestContextThreadSetupAction.java:48) at
>> io.undertow.servlet.core.ContextClassLoaderSetupAction$1.
>> call(ContextClassLoaderSetupAction.java:43) at
>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.
>> call(LegacyThreadSetupActionWrapper.java:44) at
>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.
>> call(LegacyThreadSetupActionWrapper.java:44) at
>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.
>> call(LegacyThreadSetupActionWrapper.java:44) at
>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.
>> call(LegacyThreadSetupActionWrapper.java:44) at
>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchR
>> equest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.S
>> ervletInitialHandler.access$000(ServletInitialHandler.java:81) at
>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleR
>> equest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.
>> executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerE
>> xchange$1.run(HttpServerExchange.java:805) at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoo
>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at
>> java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-12-20
>> 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand]
>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting
>> for VDSM response: Internal timeout occured
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is definitely not related to the patch linked above.
>> >>>>>>>
>> >>>>>>> However I am also not quite sure what is the cause of this right
>> now, it might be a breaking change in VDSM.
>> >>>>>>>
>> >>>>>>
>> >>>>>> The DC level is 4.0, is it possible that is the reason for the
>> failure?
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Full test code can be seen here:
>> >>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=
>> blob;f=basic-suite-4.1/test-scenarios/004_basic_sanity.py;
>> hb=refs/heads/master#l291
>> >>>>>>>>
>> >>>>>>>> Full test exception can be seen here:
>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.
>> 1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/
>> >>>>>>>>
>> >>>>>>>> Further logs can be seen in Jenkins:
>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.
>> 1/13/artifact/exported-artifacts/basic_suite_4.1.sh-el7/
>> exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Barak Korren
>> >>>>>>>> bkorren at redhat.com
>> >>>>>>>> RHCE, RHCi, RHV-DevOps Team
>> >>>>>>>> https://ifireball.wordpress.com/
>> >>>>>>>> _______________________________________________
>> >>>>>>>> Devel mailing list
>> >>>>>>>> Devel at ovirt.org
>> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> Devel mailing list
>> >>>>>>> Devel at ovirt.org
>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Eyal Edri
>> >>>>>> Associate Manager
>> >>>>>> RHV DevOps
>> >>>>>> EMEA ENG Virtualization R&D
>> >>>>>> Red Hat Israel
>> >>>>>>
>> >>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>>>>> _______________________________________________
>> >>>>>> Devel mailing list
>> >>>>>> Devel at ovirt.org
>> >>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Devel mailing list
>> >>>> Devel at ovirt.org
>> >>>> http://lists.ovirt.org/mailman/listinfo/devel
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Eyal Edri
>> > Associate Manager
>> > RHV DevOps
>> > EMEA ENG Virtualization R&D
>> > Red Hat Israel
>> >
>> > phone: +972-9-7692018 <+972%209-769-2018>
>> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >
>> > _______________________________________________
>> > Devel mailing list
>> > Devel at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>

Guys,
4.1 is broken for 11 hours now.
It seems that the issues are accumulating; looking into
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log

shows a duplicate entry in disk mapping:

2016-12-21 10:52:51,436-05 ERROR
[org.ovirt.engine.core.bll.RunVmOnceCommand]
(org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.:
java.lang.IllegalStateException: Duplicate key
org.ovirt.engine.core.common.businessentities.storage.DiskVmElement at 8f24f60b
	at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
[rt.jar:1.8.0_111]
	at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111]
	at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
[rt.jar:1.8.0_111]
	at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
[rt.jar:1.8.0_111]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
[rt.jar:1.8.0_111]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
[rt.jar:1.8.0_111]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
[rt.jar:1.8.0_111]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
[rt.jar:1.8.0_111]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[rt.jar:1.8.0_111]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
[rt.jar:1.8.0_111]
	at org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementMap(DiskHandler.java:46)
[bll.jar:]
	at org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscard(RunVmValidator.java:318)
[bll.jar:]
	at org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator.java:146)
[bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920)
[bll.jar:]
	at org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:58)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:834)
[bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388)
[bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBase.java:163)
[bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138)
[bll.jar:]
	at org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327)
[bll.jar:]


I suggest we consider reverting to the last good state, and work out
the issues from there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20161222/05e54b02/attachment-0001.html>

More information about the Devel mailing list