Re: [ovirt-devel] System tests for 4.1 currently failing to run VMs!

--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On 22 Dec 2016, at 08:57, Doron Fediuck <dfediuck@redhat.com> wrote:
>=20
>=20
>=20
> On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>=20
>> On 21 Dec 2016, at 20:52, Eyal Edri <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>>=20
>> Not as easy at it sounds,   the current flow of ost is much more =
complicated than just build artifacts. I'm not saving we won't do it,  =
but it won't be ready tomorrow and we designed lago exactly for being =
independent of any ci system so anyone can run it on their laptop.=20
>=20
> +1
> we just need to improve a little bit to make it possible for everyone. =
Almost there.
>=20
>>  So maybe using Jenkins might be simpler but people shouldn't skip =
verification just because such a job doesn't exist yet.=20
>>=20
>> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <oourfali@redhat.com =
<mailto:oourfali@redhat.com>> wrote:
>> Why not run it via Jenkins for patches?=20
>> Like, if you add a comment saying "run: ost" it will run it?=20
>>=20
>=20
> please don=E2=80=99t add more. Even the Rerun-hooks thing is hard to =
remember. Even after years I always keep asking is it "Re-run hooks=E2=80=9D=
 or =E2=80=9CRerun-Hooks=E2=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does =
case matter=E2=80=A6..bleh
> Either add the button or maybe add it as a link in the automated =
comments (that would actually work good enough)
>=20
> Thanks,
> michal
>=20
>> It do it automatically based on another thing?=20
>>=20
>> On Dec 21, 2016 17:42, "Eyal Edri" <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>> >
>> >
>> >
>> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>
>> >>
>> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul@redhat.com =
<mailto:ykaul@redhat.com>> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>>>
>> >>>>
>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>>>>
>> >>>>>
>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra =
<vfeenstr@redhat.com <mailto:vfeenstr@redhat.com>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren =
<bkorren@redhat.com <mailto:bkorren@redhat.com>> wrote:
>> >>>>>>>>
>> >>>>>>>> The test for running VMs had been failing since yesterday.
>> >>>>>>>>
>> >>>>>>>> The patch merged before the failures started was:
>> >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ =
<https://gerrit.ovirt.org/#/c/68826/>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) =
while running
>> >>>>>>>> this API call:
>> >>>>>>>>
>> >>>>>>>> api.vms.get(VM0_NAME).status.state =3D=3D =E2=80=98up'
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is a REST API call, the patch above is Frontend. So this =
is unrelated.
>> >>>>>>>
>> >>>>>>> However on Host 0 I can see this:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') The vm start process =
failed (vm:615) Traceback (most recent call last): File =
"/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm =
self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run =
self._connection.createXML(domxml, flags), File =
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, =
in wrapper ret =3D f(*args, **kwargs) File =
"/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper =
return func(inst, *args, **kwargs) File =
"/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML =
if ret is None:raise libvirtError('virDomainCreateXML() failed', =
conn=3Dself) libvirtError: internal error: process exited while =
connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: =
CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 =
15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to =
maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z =
qemu-kvm: -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: =
speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) =
to bus "usb.0", port "1" (high speed)=20
>> >>>>>
>> >>>>>
>> >>>>> it is likely related to the recent USB patches=20
>> >>>>> investigating
>> >>>>
>> >>>>
>> >>>> hm, there are multiple problems (features/bugs depending on =
prefferred point of view:)
>> >>>> but there is an easy =E2=80=9Cfix=E2=80=9D taking care of this =
particular problem, so we can start with that and figure out the proper =
approach later
>> >>>> arik will push that and merge it soon, likely today
>> >>>
>> >>>
>> >>> Thanks - if there is a quicker way to resolve this by reverting, =
I think it's a better option.
>> >>
>> >>
>> >> I really need to talk you out of this approach:-)
>> >> It does sound tempting and logical, but with our development model =
of large patch series combined with late detection it really is quite =
risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out the =
right revert patch is more complicated then fixing it.
>> >
>> >
>> > Can we start asking developers run OST before they merge so it will =
be early detection and not late detection?=20
>> > We have video sessions on how to use OST, so it shouldn't be any =
issues in running it on a patch.
>> > =20
>> >>
>> >> I believe the best is to identify it early and notify the =
maintainer who merged that patch ASAP, as that person is in the best =
position to asses if revert is safe or if there is a simple follow up =
patch he can push right away
>> >>
>> >> We can surely improve on reporting, so Barak, how/why did you =
point to that particular patch in your email? It should start failing on =
16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 =
01:56:07 2016 -0500)
>> >> Even though it may be that it was hidden because of =
c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 =
15:16:40 2016 -0500)
>> >>
>> >> (fix is ready, waiting on CI now)
>> >>
>> >> Thanks,
>> >> michal
>> >>
>> >>> Y.
>> >>> =20
>> >>>>
>> >>>>
>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: =
internal error: process exited while connecting to monitor: =
2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any =
NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: speed =
mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus =
"usb.0", port "1" (high speed) (code=3D1) (vm:1197) 2016-12-20 =
16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Stopping connection =
(guestagent:430)=20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> And on The engine loads of these:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command =
'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: =
VDSGenericException: VDSNetworkException: Timeout during rpc call =
2016-12-20 16:53:57,849-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp =
Reactor) [7971dfb4] MESSAGE content-length:80 =
destination:jms.topic.vdsm_responses content-type:application/json =
subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", =
"id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}=EF=BF=BD =
2016-12-20 16:53:57,850-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) =
[] Message received: {"jsonrpc": "2.0", "id": =
"3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 =
16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] =
(ResponseWorker) [] Not able to update response for =
"3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Timeout during rpc call at =
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo=
mmand.java:73) [vdsbroker.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos=
tSetupNetworkPoller.java:56) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet=
upNetworkPoller.java:41) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetu=
pNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCom=
mand(HostSetupNetworksCommand.java:287) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1249) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1389) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
53) [bll.jar:] at org.ovirt.engine.core.utils.tr =
<http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.execu=
teInSuppressed(TransactionSupport.java:164) [utils.jar:] at =
org.ovirt.engine.core.utils.tr =
<http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.execu=
teInScope(TransactionSupport.java:103) [utils.jar:] at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) =
[bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) =
[bll.jar:] at =
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(De=
faultBackendActionExecutor.java:13) [bll.jar:] at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] =
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) =
[bll.jar:] at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] =
at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) =
[:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] =
at =
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aro=
undInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at =
sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) =
[:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.proce=
ssInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>vocationmetrics.ExecutionTimeInter=
ceptor.processInvocation(ExecutionTimeInterceptor.java:43) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73) =
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45) =
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.si =
<http://org.jboss.as.ejb3.component.si/>ngleton.SingletonComponentInstance=
AssociationInterceptor.processInvocation(SingletonComponentInstanceAssocia=
tionInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.CurrentInvocationContex=
tInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41=
) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>vocationmetrics.WaitTimeIntercepto=
r.processInvocation(WaitTimeInterceptor.java:47) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInv=
ocation(StartupAwaitInterceptor.java:22) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.ShutDownInterceptorFact=
ory$1.processInvocation(ShutDownInterceptorFactory.java:64) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.LoggingInterceptor.proc=
essInvocation(LoggingInterceptor.java:67) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ee.component.Name =
<http://org.jboss.as.ee.component.name/>spaceContextInterceptor.processInv=
ocation(NamespaceContextInterceptor.java:50) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) =
at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636) at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) =
at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) =
at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73) at =
org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unk=
nown Source) [common.jar:] at =
org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendReso=
urce.java:250) at =
org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen=
dResource.java:182) at =
org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen=
dResource.java:170) at =
org.ovirt.engine.api.restapi.resource.BackendHostResource.se =
<http://esource.backendhostresource.se/>tupNetworks(BackendHostResource.ja=
va:212) at =
org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at =
org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.ja=
va:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =
[rt.jar:1.8.0_111] at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:=
62) [rt.jar:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:=
139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMetho=
dInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker=
.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour=
ceLocatorInvoker.java:138) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] 	at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok=
er.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour=
ceLocatorInvoker.java:133) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok=
er.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher=
.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher=
.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.servi=
ce(ServletContainerDispatcher.java:221) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht=
tpServletDispatcher.java:56) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht=
tpServletDispatcher.java:51) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
javax.servlet.http.HttpServlet.service(HttpServlet.java:790) =
[jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at =
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.j=
ava:85) at =
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav=
a:81) at =
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq=
uest(ServletSecurityRoleHandler.java:62) at =
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl=
etDispatchingHandler.java:36) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet=
InitialHandler.java:274) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletI=
nitialHandler.java:209) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatch=
erImpl.java:221) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDis=
patcherImpl.java:147) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherIm=
pl.java:111) at =
org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt=
er.java:139) at =
org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt=
er.java:68) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt=
er.java:116) at =
org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt=
er.java:71) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestAp=
iSessionMgmtFilter.java:78) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFi=
lter.java:39) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(Sso=
RestApiNegotiationFilter.java:91) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApi=
AuthFilter.java:47) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SessionValidationFilter.do =
<http://ers.sessionvalidationfilter.do/>Filter(SessionValidationFilter.jav=
a:59) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(=
RestApiSessionValidationFilter.java:35) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr=
otectionFilter.java:111) at =
org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr=
otectionFilter.java:102) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSuppo=
rtFilter.java:183) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav=
a:84) at =
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq=
uest(ServletSecurityRoleHandler.java:62) at =
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl=
etDispatchingHandler.java:36) at =
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.=
handleRequest(SecurityContextAssociationHandler.java:78) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.han=
dleRequest(SSLInformationAssociationHandler.java:131) at =
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.han=
dleRequest(ServletAuthenticationCallHandler.java:57) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.security.handlers.AuthenticationConstraintHandler.handleReques=
t(AuthenticationConstraintHandler.java:53) at =
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest=
(AbstractConfidentialityHandler.java:46) at =
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHand=
ler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at =
io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.han=
dleRequest(ServletSecurityConstraintHandler.java:59) at =
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleReques=
t(AuthenticationMechanismsHandler.java:60) at =
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.ha=
ndleRequest(CachedAuthenticatedSessionHandler.java:77) at =
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(No=
tificationReceiverHandler.java:50) at =
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.ha=
ndleRequest(AbstractSecurityContextAssociationHandler.java:43) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRe=
quest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(Serv=
letInitialHandler.java:292) at =
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletIniti=
alHandler.java:81) at =
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa=
ndler.java:138) at =
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa=
ndler.java:135) at =
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(Ser=
vletRequestContextThreadSetupAction.java:48) at =
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClass=
LoaderSetupAction.java:43) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet=
InitialHandler.java:272) at =
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletIniti=
alHandler.java:81) at =
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(Servlet=
InitialHandler.java:104) at =
io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at =
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) =
at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142) [rt.jar:1.8.0_111] at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) =
[rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM =
response: Internal timeout occured=20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is definitely not related to the patch linked above.
>> >>>>>>>
>> >>>>>>> However I am also not quite sure what is the cause of this =
right now, it might be a breaking change in VDSM.
>> >>>>>>>
>> >>>>>>
>> >>>>>> The DC level is 4.0, is it possible that is the reason for the =
failure?=20
>> >>>>>>
>> >>>>>> =20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Full test code can be seen here:
>> >>>>>>>> =
https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Dba=
sic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l2=
91 =
<https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Db=
asic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l=
291>
>> >>>>>>>>
>> >>>>>>>> Full test exception can be seen here:
>> >>>>>>>> =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepor=
t/junit/%28root%29/004_basic_sanity/vm_run/ =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepo=
rt/junit/%28root%29/004_basic_sanity/vm_run/>
>> >>>>>>>>
>> >>>>>>>> Further logs can be seen in Jenkins:
>> >>>>>>>> =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/=
exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas=
ic-suite-4.1/post-004_basic_sanity.py/ =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact=
/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba=
sic-suite-4.1/post-004_basic_sanity.py/>
>> >>>>>>>>
>> >>>>>>>> --=20
>> >>>>>>>> Barak Korren
>> >>>>>>>> bkorren@redhat.com <mailto:bkorren@redhat.com>
>> >>>>>>>> RHCE, RHCi, RHV-DevOps Team
>> >>>>>>>> https://ifireball.wordpress.com/ =
<https://ifireball.wordpress.com/>
>> >>>>>>>> _______________________________________________
>> >>>>>>>> Devel mailing list
>> >>>>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> Devel mailing list
>> >>>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --=20
>> >>>>>> Eyal Edri
>> >>>>>> Associate Manager
>> >>>>>> RHV DevOps
>> >>>>>> EMEA ENG Virtualization R&D
>> >>>>>> Red Hat Israel
>> >>>>>>
>> >>>>>> phone: +972-9-7692018 <tel:+972%209-769-2018>
>> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>>>>> _______________________________________________
>> >>>>>> Devel mailing list
>> >>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Devel mailing list
>> >>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --=20
>> > Eyal Edri
>> > Associate Manager
>> > RHV DevOps
>> > EMEA ENG Virtualization R&D
>> > Red Hat Israel
>> >
>> > phone: +972-9-7692018 <tel:+972%209-769-2018>
>> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >
>> > _______________________________________________
>> > Devel mailing list
>> > Devel@ovirt.org <mailto:Devel@ovirt.org>
>> > http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>=20
>=20
> _______________________________________________
> Devel mailing list
> Devel@ovirt.org <mailto:Devel@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>=20
> Guys,
> 4.1 is broken for 11 hours now.
> It seems that the issues are accumulating; looking into =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/=
exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas=
ic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log=
_ovirt-engine/engine.log =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact=
/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba=
sic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_lo=
g_ovirt-engine/engine.log>
>=20
> shows a duplicate entry in disk mapping:

interesting error, Arik. Reruns should be added to lago
The root cause is few lines above and the same as was fixed yesterday on =
master. It just needs a backport to 4.1=20
> 2016-12-21 10:52:51,436-05 ERROR =
[org.ovirt.engine.core.bll.RunVmOnceCommand] =
(org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: =
java.lang.IllegalStateException: Duplicate key =
org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6=
0b
> 	at =
java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) =
[rt.jar:1.8.0_111]
> 	at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) =
[rt.jar:1.8.0_111]
> 	at =
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1=
374) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47=
1) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)=
 [rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) =
[rt.jar:1.8.0_111]
> 	at =
org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM=
ap(DiskHandler.java:46) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar=
d(RunVmValidator.java:318) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator=
.java:146) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) =
[bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:=
58) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83=
4) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) =
[bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa=
se.java:163) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138=
) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) =
[bll.jar:]
>=20
> I suggest we consider reverting to the last good state, and work out =
the issues from there.


--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div =
class=3D"">On 22 Dec 2016, at 08:57, Doron Fediuck <<a =
href=3D"mailto:dfediuck@redhat.com" class=3D"">dfediuck@redhat.com</a>>=
 wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div =
dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On Thu, Dec 22, 2016 at 9:48 AM, =
Michal Skrivanek <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>></span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
style=3D"word-wrap:break-word" class=3D""><br class=3D""><div =
class=3D""><span class=3D"gmail-"><blockquote type=3D"cite" =
class=3D""><div class=3D"">On 21 Dec 2016, at 20:52, Eyal Edri <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:</div><br =
class=3D"gmail-m_-6974087903737897314Apple-interchange-newline"><div =
class=3D""><div dir=3D"auto" class=3D"">Not as easy at it sounds,   =
the current flow of ost is much more complicated than just build =
artifacts. I'm not saving we won't do it,  but it won't be ready =
tomorrow and we designed lago exactly for being independent of any ci =
system so anyone can run it on their =
laptop. </div></div></blockquote><div class=3D""><br =
class=3D""></div></span>+1</div><div class=3D"">we just need to improve =
a little bit to make it possible for everyone. Almost there.</div><div =
class=3D""><span class=3D"gmail-"><br class=3D""><blockquote type=3D"cite"=
 class=3D""><div class=3D""><div dir=3D"auto" class=3D""><div dir=3D"auto"=
 class=3D""> So maybe using Jenkins might be simpler but people =
shouldn't skip verification just because such a job doesn't exist =
yet. </div></div><div class=3D"gmail_extra"><br class=3D""><div =
class=3D"gmail_quote">On Dec 21, 2016 9:02 PM, "Oved Ourfali" <<a =
href=3D"mailto:oourfali@redhat.com" target=3D"_blank" =
class=3D"">oourfali@redhat.com</a>> wrote:<br type=3D"attribution" =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p =
dir=3D"ltr" class=3D"">Why not run it via Jenkins for patches? <br =
class=3D"">
Like, if you add a comment saying "run: ost" it will run it? <br =
class=3D""></p></blockquote></div></div></div></blockquote><div =
class=3D""><br class=3D""></div></span>please don=E2=80=99t add more. =
Even the Rerun-hooks thing is hard to remember. Even after years I =
always keep asking is it "Re-run hooks=E2=80=9D or =E2=80=9CRerun-Hooks=E2=
=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does case =
matter=E2=80=A6..bleh</div><div class=3D"">Either add the button or =
maybe add it as a link in the automated comments (that would actually =
work good enough)</div><div class=3D""><br class=3D""></div><div =
class=3D"">Thanks,</div><div class=3D"">michal<div class=3D""><div =
class=3D"gmail-h5"><br class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><div class=3D"gmail_extra"><div =
class=3D"gmail_quote"><blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex"><p dir=3D"ltr" class=3D"">
It do it automatically based on another thing? <br class=3D""></p><p =
dir=3D"ltr" class=3D"">On Dec 21, 2016 17:42, "Eyal Edri" <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:<br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
> On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>><br class=3D"">
>><br class=3D"">
>>> On 21 Dec 2016, at 16:25, Yaniv Kaul <<a =
href=3D"mailto:ykaul@redhat.com" target=3D"_blank" =
class=3D"">ykaul@redhat.com</a>> wrote:<br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:<br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz =
Feenstra <<a href=3D"mailto:vfeenstr@redhat.com" target=3D"_blank" =
class=3D"">vfeenstr@redhat.com</a>> wrote:<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak =
Korren <<a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" =
class=3D"">bkorren@redhat.com</a>> wrote:<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The test for running VMs had been =
failing since yesterday.<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The patch merged before the failures =
started was:<br class=3D"">
>>>>>>>> <a =
href=3D"https://gerrit.ovirt.org/#/c/68826/" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/#/c/6<wbr class=3D"">8826/</a><br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The error we`re seeing is a time-out =
(after two minutes) while running<br class=3D"">
>>>>>>>> this API call:<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> api.vms.get(VM0_NAME).status.s<wbr =
class=3D"">tate =3D=3D =E2=80=98up'<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> This is a REST API call, the patch above is =
Frontend. So this is unrelated.<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> However on Host 0 I can see this:<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) =
[virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') =
The vm start process failed (vm:615) Traceback (most recent call last): =
File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm =
self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run =
self._connection.createXML(dom<wbr class=3D"">xml, flags), File =
"/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/vdsm/libvirtconnection.py"<wbr class=3D"">, line 123, in =
wrapper ret =3D f(*args, **kwargs) File =
"/usr/lib/python2.7/site-packa<wbr class=3D"">ges/vdsm/utils.py", line =
941, in wrapper return func(inst, *args, **kwargs) File =
"/usr/lib64/python2.7/site-pac<wbr class=3D"">kages/libvirt.py", line =
3782, in createXML if ret is None:raise =
libvirtError('virDomainCreateX<wbr class=3D"">ML() failed', conn=3Dself) =
libvirtError: internal error: process exited while connecting to =
monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not =
present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: =
Warning: speed mismatch trying to attach usb device "QEMU USB CCID" =
(full speed) to bus "usb.0", port "1" (high speed) <br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>>> it is likely related to the recent USB =
patches <br class=3D"">
>>>>> investigating<br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>> hm, there are multiple problems (features/bugs =
depending on prefferred point of view:)<br class=3D"">
>>>> but there is an easy =E2=80=9Cfix=E2=80=9D taking care =
of this particular problem, so we can start with that and figure out the =
proper approach later<br class=3D"">
>>>> arik will push that and merge it soon, likely today<br =
class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>> Thanks - if there is a quicker way to resolve this by =
reverting, I think it's a better option.<br class=3D"">
>><br class=3D"">
>><br class=3D"">
>> I really need to talk you out of this approach:-)<br class=3D"">
>> It does sound tempting and logical, but with our development =
model of large patch series combined with late detection it really is =
quite risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out =
the right revert patch is more complicated then fixing it.<br class=3D"">
><br class=3D"">
><br class=3D"">
> Can we start asking developers run OST before they merge so it will =
be early detection and not late detection? <br class=3D"">
> We have video sessions on how to use OST, so it shouldn't be any =
issues in running it on a patch.<br class=3D"">
>  <br class=3D"">
>><br class=3D"">
>> I believe the best is to identify it early and notify the =
maintainer who merged that patch ASAP, as that person is in the best =
position to asses if revert is safe or if there is a simple follow up =
patch he can push right away<br class=3D"">
>><br class=3D"">
>> We can surely improve on reporting, so Barak, how/why did you =
point to that particular patch in your email? It should start failing on =
16c2ec236184b3152f1df8e874b431<wbr class=3D"">15f78d0989 (CommitDate: =
Fri Dec 16 01:56:07 2016 -0500)<br class=3D"">
>> Even though it may be that it was hidden because =
of c46f653a7846c3c2a76507b8dcf<wbr class=3D"">5bc0391ec5709 =
(CommitDate: Mon Dec 19 15:16:40 2016 -0500)<br class=3D"">
>><br class=3D"">
>> (fix is ready, waiting on CI now)<br class=3D"">
>><br class=3D"">
>> Thanks,<br class=3D"">
>> michal<br class=3D"">
>><br class=3D"">
>>> Y.<br class=3D"">
>>>  <br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) =
[virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') =
Changed state to Down: internal error: process exited while connecting =
to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not =
present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: =
Warning: speed mismatch trying to attach usb device "QEMU USB CCID" =
(full speed) to bus "usb.0", port "1" (high speed) (code=3D1) (vm:1197) =
2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') Stopping =
connection (guestagent:430) <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> And on The engine loads of these:<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>> 2016-12-20 16:53:57,844-05 ERROR =
[org.ovirt.engine.core.vdsbrok<wbr class=3D"">er.vdsbroker.PollVDSCommand]=
 (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] =
Command 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, =
VdsIdVDSCommandParametersBase:<wbr class=3D"">{runAsync=3D'true', =
hostId=3D'994b5d79-605f-4415-94f<wbr class=3D"">2-02c79cfa246e'})' =
execution failed: VDSGenericException: VDSNetworkException: Timeout =
during rpc call 2016-12-20 16:53:57,849-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.reactors.stomp.impl.Message] (SSL Stomp Reactor) [7971dfb4] =
MESSAGE content-length:80 destination:jms.topic.vdsm_res<wbr =
class=3D"">ponses content-type:application/json =
subscription:5b6494d5-d5a0-477<wbr class=3D"">1-941c-a8be70f72450 =
{"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr =
class=3D"">95c122d", "result": true}=EF=BF=BD 2016-12-20 16:53:57,850-05 =
DEBUG [org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.internal.ResponseWorker] (ResponseWorker) [] Message =
received: {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr =
class=3D"">95c122d", "result": true} 2016-12-20 16:53:57,850-05 ERROR =
[org.ovirt.vdsm.jsonrpc.client<wbr class=3D"">.JsonRpcClient] =
(ResponseWorker) [] Not able to update response for =
"3c95fdb0-5b77-4927-9f6e-adc73<wbr class=3D"">95c122d" 2016-12-20 =
16:53:57,844-05 DEBUG [org.ovirt.engine.core.vdsbrok<wbr =
class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) =
[5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Exception: =
org.ovirt.engine.core.vdsbroke<wbr =
class=3D"">r.vdsbroker.VDSNetworkExceptio<wbr class=3D"">n: =
VDSGenericException: VDSNetworkException: Timeout during rpc call 	=
at org.ovirt.engine.core.vdsbroke<wbr =
class=3D"">r.vdsbroker.FutureVDSCommand.<wbr =
class=3D"">get(FutureVDSCommand.java:73) [vdsbroker.jar:] 	at =
org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworkPolle<wbr =
class=3D"">r.getValue(HostSetupNetworkPol<wbr class=3D"">ler.java:56) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworkPolle<wbr =
class=3D"">r.poll(HostSetupNetworkPoller.<wbr class=3D"">java:41) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworksComm<wbr =
class=3D"">and.invokeSetupNetworksCommand<wbr =
class=3D"">(HostSetupNetworksCommand.<wbr class=3D"">java:426) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworksComm<wbr =
class=3D"">and.executeCommand(HostSetupNe<wbr =
class=3D"">tworksCommand.java:287) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeWithoutTransact<wbr =
class=3D"">ion(CommandBase.java:1249) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeActionInTransac<wbr =
class=3D"">tionScope(CommandBase.java:<wbr class=3D"">1389) [bll.jar:] 	=
at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.runInTransaction(Comma<wbr class=3D"">ndBase.java:2053)=
 [bll.jar:] 	at <a href=3D"http://org.ovirt.engine.core.utils.tr/" =
class=3D"">org.ovirt.engine.core.utils.tr</a><wbr =
class=3D"">ansaction.TransactionSupport.e<wbr =
class=3D"">xecuteInSuppressed(Transaction<wbr class=3D"">Support.java:164)=
 [utils.jar:] 	at <a href=3D"http://org.ovirt.engine.core.utils.tr/" =
class=3D"">org.ovirt.engine.core.utils.tr</a><wbr =
class=3D"">ansaction.TransactionSupport.e<wbr =
class=3D"">xecuteInScope(TransactionSuppo<wbr class=3D"">rt.java:103) =
[utils.jar:] 	at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.execute(CommandBase.<wbr class=3D"">java:1449) =
[bll.jar:] 	at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeAction(CommandB<wbr class=3D"">ase.java:395) =
[bll.jar:] 	at org.ovirt.engine.core.bll.exec<wbr =
class=3D"">utor.DefaultBackendActionExecu<wbr =
class=3D"">tor.execute(DefaultBackendActi<wbr =
class=3D"">onExecutor.java:13) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runAction(Backend.java:<wbr class=3D"">511) [bll.jar:] 	=
at org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runActionImpl(Backend.java<wbr class=3D"">:493) =
[bll.jar:] 	at org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runAction(Backend.java:<wbr class=3D"">446) [bll.jar:] 	=
at sun.reflect.GeneratedMethodAcc<wbr class=3D"">essor232.invoke(Unknown =
Source) [:1.8.0_111] 	at sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.as.ee.component.Mana<wbr =
class=3D"">gedReferenceMethodInterceptor.<wbr =
class=3D"">processInvocation(ManagedRefer<wbr =
class=3D"">enceMethodInterceptor.java:52) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.delegateInte<wbr =
class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:70) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.doMethodInte<wbr =
class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:80) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.processInvoc<wbr =
class=3D"">ation(Jsr299BindingsIntercepto<wbr class=3D"">r.java:93) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ee.component.inte<wbr =
class=3D"">rceptors.UserInterceptorFactor<wbr =
class=3D"">y$1.processInvocation(UserInte<wbr =
class=3D"">rceptorFactory.java:63) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.ovirt.engine.core.bll.inte<wbr =
class=3D"">rceptors.CorrelationIdTrackerI<wbr =
class=3D"">nterceptor.aroundInvoke(Correl<wbr =
class=3D"">ationIdTrackerInterceptor.<wbr class=3D"">java:13) [bll.jar:] =
	at sun.reflect.GeneratedMethodAcc<wbr =
class=3D"">essor64.invoke(Unknown Source) [:1.8.0_111] 	at =
sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.as.ee.component.Mana<wbr =
class=3D"">gedReferenceLifecycleMethodInt<wbr =
class=3D"">erceptor.processInvocation(Man<wbr =
class=3D"">agedReferenceLifecycleMethodIn<wbr =
class=3D"">terceptor.java:89) 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">vocationmetrics.ExecutionTimeI<wbr =
class=3D"">nterceptor.processInvocation(E<wbr =
class=3D"">xecutionTimeInterceptor.java:4<wbr class=3D"">3) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.jboss.weld.ejb.AbstractEJB<wbr =
class=3D"">RequestScopeActivationIntercep<wbr =
class=3D"">tor.aroundInvoke(AbstractEJBRe<wbr =
class=3D"">questScopeActivationIntercepto<wbr class=3D"">r.java:73) =
[weld-core-impl-2.3.5.Final.ja<wbr class=3D"">r:2.3.5.Final] 	at =
org.jboss.as.weld.ejb.EjbReque<wbr =
class=3D"">stScopeActivationInterceptor.<wbr =
class=3D"">processInvocation(EjbRequestSc<wbr =
class=3D"">opeActivationInterceptor.java:<wbr class=3D"">83) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ee.concurrent.Con<wbr =
class=3D"">currentContextInterceptor.proc<wbr =
class=3D"">essInvocation(ConcurrentContex<wbr =
class=3D"">tInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:1<wbr =
class=3D"">0.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.InitialIn<wbr =
class=3D"">terceptor.processInvocation(In<wbr =
class=3D"">itialInterceptor.java:21) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.inte<wbr =
class=3D"">rceptors.ComponentDispatcherIn<wbr =
class=3D"">terceptor.processInvocation(Co<wbr =
class=3D"">mponentDispatcherInterceptor.<wbr class=3D"">java:52) 	=
at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.si/" =
class=3D"">org.jboss.as.ejb3.component.si</a><wbr =
class=3D"">ngleton.SingletonComponentInst<wbr =
class=3D"">anceAssociationInterceptor.pro<wbr =
class=3D"">cessInvocation(SingletonCompon<wbr =
class=3D"">entInstanceAssociationIntercep<wbr class=3D"">tor.java:53) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.invokeInNoTx(CMTTxInte<wbr class=3D"">rceptor.java:263)=
 [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.supports(CMTTxIntercep<wbr class=3D"">tor.java:374) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.processInvocation(CMTT<wbr =
class=3D"">xInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.CurrentInvocationCo<wbr =
class=3D"">ntextInterceptor.<wbr =
class=3D"">processInvocation(CurrentInvoc<wbr =
class=3D"">ationContextInterceptor.java:<wbr class=3D"">41) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">vocationmetrics.WaitTimeInterc<wbr =
class=3D"">eptor.processInvocation(WaitTi<wbr =
class=3D"">meInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.security.Sec<wbr =
class=3D"">urityContextInterceptor.proces<wbr =
class=3D"">sInvocation(SecurityContextInt<wbr =
class=3D"">erceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.deployment.p<wbr =
class=3D"">rocessors.StartupAwaitIntercep<wbr =
class=3D"">tor.processInvocation(StartupA<wbr =
class=3D"">waitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.ShutDownInterceptor<wbr =
class=3D"">Factory$1.processInvocation(Sh<wbr =
class=3D"">utDownInterceptorFactory.java:<wbr class=3D"">64) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.LoggingInterceptor.<wbr =
class=3D"">processInvocation(LoggingInter<wbr class=3D"">ceptor.java:67) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ee.component.name/" =
class=3D"">org.jboss.as.ee.component.Name</a><wbr =
class=3D"">spaceContextInterceptor.proces<wbr =
class=3D"">sInvocation(NamespaceContextIn<wbr =
class=3D"">terceptor.java:50) 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ContextCl<wbr =
class=3D"">assLoaderInterceptor.processIn<wbr =
class=3D"">vocation(ContextClassLoaderInt<wbr class=3D"">erceptor.java:64)=
 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) 	=
at org.wildfly.security.manager.W<wbr =
class=3D"">ildFlySecurityManager.doChecke<wbr =
class=3D"">d(WildFlySecurityManager.java:<wbr class=3D"">636) 	at =
org.jboss.invocation.AccessChe<wbr =
class=3D"">ckingInterceptor.processInvoca<wbr =
class=3D"">tion(AccessCheckingInterceptor<wbr class=3D"">.java:61) 	=
at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) 	=
at org.jboss.invocation.Privilege<wbr =
class=3D"">dWithCombinerInterceptor.<wbr =
class=3D"">processInvocation(PrivilegedWi<wbr =
class=3D"">thCombinerInterceptor.java:80) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.View<wbr =
class=3D"">Service$View.invoke(ViewServic<wbr class=3D"">e.java:198) 	=
at org.jboss.as.ee.component.View<wbr =
class=3D"">Description$1.processInvocatio<wbr =
class=3D"">n(ViewDescription.java:185) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.Prox<wbr =
class=3D"">yInvocationHandler.invoke(Prox<wbr =
class=3D"">yInvocationHandler.java:73) 	at =
org.ovirt.engine.core.common.i<wbr =
class=3D"">nterfaces.BackendLocal$$$view4<wbr =
class=3D"">.runAction(Unknown Source) [common.jar:] 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.doActi<wbr =
class=3D"">on(BackendResource.java:250) 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.perfor<wbr =
class=3D"">mAction(BackendResource.java:<wbr class=3D"">182) 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.perfor<wbr =
class=3D"">mAction(BackendResource.java:<wbr class=3D"">170) 	at =
org.ovirt.engine.api.restapi.r<wbr class=3D""><a =
href=3D"http://esource.backendhostresource.se/" =
class=3D"">esource.BackendHostResource.se</a><wbr =
class=3D"">tupNetworks(BackendHostResourc<wbr class=3D"">e.java:212) 	=
at org.ovirt.engine.api.v3.V3Serv<wbr =
class=3D"">er.adaptAction(V3Server.java:<wbr class=3D"">216) 	at =
org.ovirt.engine.api.v3.server<wbr =
class=3D"">s.V3HostServer.setupNetworks(<wbr =
class=3D"">V3HostServer.java:182) 	at =
sun.reflect.NativeMethodAccess<wbr class=3D"">orImpl.invoke0(Native =
Method) [rt.jar:1.8.0_111] 	at sun.reflect.NativeMethodAccess<wbr =
class=3D"">orImpl.invoke(NativeMethodAcce<wbr class=3D"">ssorImpl.java:62)=
 [rt.jar:1.8.0_111] 	at sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.resteasy.core.Method<wbr =
class=3D"">InjectorImpl.invoke(MethodInje<wbr =
class=3D"">ctorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceMethodInvoker.invokeOnTarget<wbr =
class=3D"">(ResourceMethodInvoker.java:29<wbr class=3D"">5) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceMethodInvoker.invoke(Resourc<wbr =
class=3D"">eMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invokeOnTarge<wbr =
class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:138) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invoke(Resour<wbr =
class=3D"">ceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invokeOnTarge<wbr =
class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:133) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invoke(Resour<wbr =
class=3D"">ceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Synchr<wbr =
class=3D"">onousDispatcher.invoke(Synchro<wbr =
class=3D"">nousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Synchr<wbr =
class=3D"">onousDispatcher.invoke(Synchro<wbr =
class=3D"">nousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.ServletContainerDi<wbr =
class=3D"">spatcher.service(ServletContai<wbr =
class=3D"">nerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.HttpServletDispatc<wbr =
class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:56) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.HttpServletDispatc<wbr =
class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:51) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
javax.servlet.http.HttpServlet<wbr =
class=3D"">.service(HttpServlet.java:790) =
[jboss-servlet-api_3.1_spec-1.<wbr class=3D"">0.0.Final.jar:1.0.0.Final] =
	at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletHandler.handleRequest(Se<wbr =
class=3D"">rvletHandler.java:85) 	at =
io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler.handleRequest(Fil<wbr =
class=3D"">terHandler.java:81) 	at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityRoleHan<wbr =
class=3D"">dler.handleRequest(ServletSecu<wbr =
class=3D"">rityRoleHandler.java:62) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletDispatchingHandler.handl<wbr =
class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:274) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchT<wbr =
class=3D"">oPath(ServletInitialHandler.<wbr class=3D"">java:209) 	=
at io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forwardImpl(R<wbr =
class=3D"">equestDispatcherImpl.java:221) 	at =
io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forwardImplSe<wbr =
class=3D"">tup(RequestDispatcherImpl.<wbr class=3D"">java:147) 	at =
io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forward(Reque<wbr =
class=3D"">stDispatcherImpl.java:111) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.VersionFilter.doFilt<wbr =
class=3D"">er(VersionFilter.java:139) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.VersionFilter.doFilt<wbr =
class=3D"">er(VersionFilter.java:68) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.CurrentFilter.doFilt<wbr =
class=3D"">er(CurrentFilter.java:116) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.CurrentFilter.doFilt<wbr =
class=3D"">er(CurrentFilter.java:71) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.RestApiSessionMgmtFilter.d<wbr =
class=3D"">oFilter(RestApiSessionMgmtFilt<wbr class=3D"">er.java:78) =
[aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.EnforceAuthFilter.doFilter<wbr =
class=3D"">(EnforceAuthFilter.java:39) [aaa.jar:] 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.SsoRestApiNegotiationFilte<wbr =
class=3D"">r.doFilter(SsoRestApiNegotiati<wbr class=3D"">onFilter.java:91)=
 [aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.SsoRestApiAuthFilter.doFil<wbr =
class=3D"">ter(SsoRestApiAuthFilter.java:<wbr class=3D"">47) [aaa.jar:] 	=
at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr class=3D""><a =
href=3D"http://ers.sessionvalidationfilter.do/" =
class=3D"">ers.SessionValidationFilter.do</a><wbr =
class=3D"">Filter(SessionValidationFilter<wbr class=3D"">.java:59) =
[aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.RestApiSessionValidationFi<wbr =
class=3D"">lter.doFilter(RestApiSessionVa<wbr =
class=3D"">lidationFilter.java:35) [aaa.jar:] 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CSRFProtectionFilter.d<wbr =
class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:111) 	=
at org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CSRFProtectionFilter.d<wbr =
class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:102) 	=
at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CORSSupportFilter.doFi<wbr =
class=3D"">lter(CORSSupportFilter.java:<wbr class=3D"">183) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler.handleRequest(Fil<wbr =
class=3D"">terHandler.java:84) 	at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityRoleHan<wbr =
class=3D"">dler.handleRequest(ServletSecu<wbr =
class=3D"">rityRoleHandler.java:62) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletDispatchingHandler.handl<wbr =
class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) 	=
at org.wildfly.extension.undertow<wbr =
class=3D"">.security.SecurityContextAssoc<wbr =
class=3D"">iationHandler.handleRequest(Se<wbr =
class=3D"">curityContextAssociationHandle<wbr class=3D"">r.java:78) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.SSLInformationAssociat<wbr =
class=3D"">ionHandler.handleRequest(SSLIn<wbr =
class=3D"">formationAssociationHandler.<wbr class=3D"">java:131) 	=
at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletAuthenticationC<wbr =
class=3D"">allHandler.handleRequest(Servl<wbr =
class=3D"">etAuthenticationCallHandler.<wbr class=3D"">java:57) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.security.handlers.<wbr =
class=3D"">AuthenticationConstraintHandle<wbr =
class=3D"">r.handleRequest(Authentication<wbr =
class=3D"">ConstraintHandler.java:53) 	at =
io.undertow.security.handlers.<wbr =
class=3D"">AbstractConfidentialityHandler<wbr =
class=3D"">.handleRequest(AbstractConfide<wbr =
class=3D"">ntialityHandler.java:46) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletConfidentiality<wbr =
class=3D"">ConstraintHandler.<wbr =
class=3D"">handleRequest(ServletConfident<wbr =
class=3D"">ialityConstraintHandler.java:<wbr class=3D"">64) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityConstra<wbr =
class=3D"">intHandler.handleRequest(Servl<wbr =
class=3D"">etSecurityConstraintHandler.<wbr class=3D"">java:59) 	=
at io.undertow.security.handlers.<wbr =
class=3D"">AuthenticationMechanismsHandle<wbr =
class=3D"">r.handleRequest(Authentication<wbr =
class=3D"">MechanismsHandler.java:60) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.CachedAuthenticatedSes<wbr =
class=3D"">sionHandler.handleRequest(Cach<wbr =
class=3D"">edAuthenticatedSessionHandler.<wbr class=3D"">java:77) 	=
at io.undertow.security.handlers.<wbr =
class=3D"">NotificationReceiverHandler.ha<wbr =
class=3D"">ndleRequest(NotificationReceiv<wbr =
class=3D"">erHandler.java:50) 	at io.undertow.security.handlers.<wbr =
class=3D"">AbstractSecurityContextAssocia<wbr =
class=3D"">tionHandler.handleRequest(Abst<wbr =
class=3D"">ractSecurityContextAssociation<wbr class=3D"">Handler.java:43) =
	at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
org.wildfly.extension.undertow<wbr =
class=3D"">.security.jacc.JACCContextIdHa<wbr =
class=3D"">ndler.handleRequest(JACCContex<wbr =
class=3D"">tIdHandler.java:61) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.handleFir<wbr =
class=3D"">stRequest(ServletInitialHandle<wbr class=3D"">r.java:292) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.access$10<wbr =
class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$2.call(Se<wbr =
class=3D"">rvletInitialHandler.java:138) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$2.call(Se<wbr =
class=3D"">rvletInitialHandler.java:135) 	at =
io.undertow.servlet.core.Servl<wbr =
class=3D"">etRequestContextThreadSetupAct<wbr =
class=3D"">ion$1.call(ServletRequestConte<wbr =
class=3D"">xtThreadSetupAction.java:48) 	at =
io.undertow.servlet.core.Conte<wbr =
class=3D"">xtClassLoaderSetupAction$1.<wbr =
class=3D"">call(ContextClassLoaderSetupAc<wbr class=3D"">tion.java:43) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:272) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.access$00<wbr =
class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$1.handleR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:104) 	=
at io.undertow.server.Connectors.<wbr =
class=3D"">executeRootHandler(Connectors.<wbr class=3D"">java:202) 	=
at io.undertow.server.HttpServerE<wbr =
class=3D"">xchange$1.run(HttpServerExchan<wbr class=3D"">ge.java:805) 	=
at java.util.concurrent.ThreadPoo<wbr =
class=3D"">lExecutor.runWorker(ThreadPool<wbr =
class=3D"">Executor.java:1142) [rt.jar:1.8.0_111] 	at =
java.util.concurrent.ThreadPoo<wbr =
class=3D"">lExecutor$Worker.run(ThreadPoo<wbr =
class=3D"">lExecutor.java:617) [rt.jar:1.8.0_111] 	at =
java.lang.Thread.run(Thread.ja<wbr class=3D"">va:745) [rt.jar:1.8.0_111] =
2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbrok<wbr =
class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) =
[5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Timeout waiting =
for VDSM response: Internal timeout occured <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> This is definitely not related to the patch =
linked above.<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> However I am also not quite sure what is =
the cause of this right now, it might be a breaking change in VDSM.<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> The DC level is 4.0, is it possible that is the =
reason for the failure? <br class=3D"">
>>>>>><br class=3D"">
>>>>>>  <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Full test code can be seen here:<br =
class=3D"">
>>>>>>>> <a =
href=3D"https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblo=
b;f=3Dbasic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/m=
aster#l291" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/gitwe<wbr =
class=3D"">b?p=3Dovirt-system-tests.git;a=3D<wbr =
class=3D"">blob;f=3Dbasic-suite-4.1/test-<wbr =
class=3D"">scenarios/004_basic_sanity.py;<wbr =
class=3D"">hb=3Drefs/heads/master#l291</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Full test exception can be seen =
here:<br class=3D"">
>>>>>>>> <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/t=
estReport/junit/%28root%29/004_basic_sanity/vm_run/" target=3D"_blank" =
class=3D"">http://jenkins.ovirt.org/job/t<wbr =
class=3D"">est-repo_ovirt_experimental_4.<wbr =
class=3D"">1/13/testReport/junit/%28root%<wbr =
class=3D"">29/004_basic_sanity/vm_run/</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Further logs can be seen in Jenkins:<br =
class=3D"">
>>>>>>>> <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/a=
rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_=
logs/basic-suite-4.1/post-004_basic_sanity.py/" target=3D"_blank" =
class=3D"">http://jenkins.ovirt.org/job/t<wbr =
class=3D"">est-repo_ovirt_experimental_4.<wbr =
class=3D"">1/13/artifact/exported-artifac<wbr =
class=3D"">ts/basic_suite_4.1.sh-el7/<wbr =
class=3D"">exported-artifacts/test_logs/<wbr =
class=3D"">basic-suite-4.1/post-004_basic<wbr =
class=3D"">_sanity.py/</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> -- <br class=3D"">
>>>>>>>> Barak Korren<br class=3D"">
>>>>>>>> <a href=3D"mailto:bkorren@redhat.com" =
target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">
>>>>>>>> RHCE, RHCi, RHV-DevOps Team<br =
class=3D"">
>>>>>>>> <a =
href=3D"https://ifireball.wordpress.com/" target=3D"_blank" =
class=3D"">https://ifireball.wordpress.co<wbr class=3D"">m/</a><br =
class=3D"">
>>>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>>>> Devel mailing list<br class=3D"">
>>>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>>> Devel mailing list<br class=3D"">
>>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> -- <br class=3D"">
>>>>>> Eyal Edri<br class=3D"">
>>>>>> Associate Manager<br class=3D"">
>>>>>> RHV DevOps<br class=3D"">
>>>>>> EMEA ENG Virtualization R&D<br class=3D"">
>>>>>> Red Hat Israel<br class=3D"">
>>>>>><br class=3D"">
>>>>>> phone: <a href=3D"tel:+972%209-769-2018" =
value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br =
class=3D"">
>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br =
class=3D"">
>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>> Devel mailing list<br class=3D"">
>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>> Devel mailing list<br class=3D"">
>>>> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel"=
 target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>><br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
> -- <br class=3D"">
> Eyal Edri<br class=3D"">
> Associate Manager<br class=3D"">
> RHV DevOps<br class=3D"">
> EMEA ENG Virtualization R&D<br class=3D"">
> Red Hat Israel<br class=3D"">
><br class=3D"">
> phone: <a href=3D"tel:+972%209-769-2018" value=3D"+97297692018" =
target=3D"_blank" class=3D"">+972-9-7692018</a><br class=3D"">
> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br class=3D"">
><br class=3D"">
> ______________________________<wbr class=3D"">_________________<br =
class=3D"">
> Devel mailing list<br class=3D"">
> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D""></p>
</blockquote></div></div>
</div></blockquote></div></div></div><br class=3D""></div><br =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">
Devel mailing list<br class=3D"">
<a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br =
class=3D"">
<a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br=
 class=3D""></div><div class=3D"gmail_extra">Guys,</div><div =
class=3D"gmail_extra">4.1 is broken for 11 hours now.</div><div =
class=3D"gmail_extra">It seems that the issues are accumulating; looking =
into <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/a=
rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_=
logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/=
_var_log_ovirt-engine/engine.log" =
class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/2=
8/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/te=
st_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engi=
ne/_var_log_ovirt-engine/engine.log</a></div><div =
class=3D"gmail_extra"><br class=3D""></div><div =
class=3D"gmail_extra">shows a duplicate entry in disk =
mapping:</div></div></div></blockquote><div><br =
class=3D""></div>interesting error, Arik. Reruns should be added to =
lago</div><div>The root cause is few lines above and the same as was =
fixed yesterday on master. It just needs a backport to 4.1 <br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><pre style=3D"" =
class=3D"">2016-12-21 10:52:51,436-05 ERROR =
[org.ovirt.engine.core.bll.RunVmOnceCommand] =
(org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: =
java.lang.IllegalStateException: Duplicate key =
org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6=
0b
	at =
java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) =
[rt.jar:1.8.0_111]
	at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111]
	at =
java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) =
[rt.jar:1.8.0_111]
	at =
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1=
374) [rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47=
1) [rt.jar:1.8.0_111]
	at =
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)=
 [rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) =
[rt.jar:1.8.0_111]
	at =
org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM=
ap(DiskHandler.java:46) [bll.jar:]
	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar=
d(RunVmValidator.java:318) [bll.jar:]
	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator=
.java:146) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) =
[bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:=
58) [bll.jar:]
	at =
org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83=
4) [bll.jar:]
	at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) =
[bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa=
se.java:163) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138=
) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) =
[bll.jar:]</pre><pre style=3D"" class=3D""><br class=3D""></pre><pre =
style=3D"" class=3D"">I suggest we consider reverting to the last good =
state, and work out the issues from there.</pre></div></div>
</div></blockquote></div><br class=3D""></body></html>=

--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3--