New subject: System tests for 4.1 currently failing to run VMs!

21 Dec 2016

--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On 22 Dec 2016, at 08:57, Doron Fediuck <dfediuck@redhat.com> wrote:
>=20
>=20
>=20
> On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>=20
>> On 21 Dec 2016, at 20:52, Eyal Edri <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>>=20
>> Not as easy at it sounds,   the current flow of ost is much more =
complicated than just build artifacts. I'm not saving we won't do it,  =
but it won't be ready tomorrow and we designed lago exactly for being =
independent of any ci system so anyone can run it on their laptop.=20
>=20
> +1
> we just need to improve a little bit to make it possible for everyone. =
Almost there.
>=20
>>  So maybe using Jenkins might be simpler but people shouldn't skip =
verification just because such a job doesn't exist yet.=20
>>=20
>> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <oourfali@redhat.com =
<mailto:oourfali@redhat.com>> wrote:
>> Why not run it via Jenkins for patches?=20
>> Like, if you add a comment saying "run: ost" it will run it?=20
>>=20
>=20
> please don=E2=80=99t add more. Even the Rerun-hooks thing is hard to =
remember. Even after years I always keep asking is it "Re-run hooks=E2=80=9D=
 or =E2=80=9CRerun-Hooks=E2=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does =
case matter=E2=80=A6..bleh
> Either add the button or maybe add it as a link in the automated =
comments (that would actually work good enough)
>=20
> Thanks,
> michal
>=20
>> It do it automatically based on another thing?=20
>>=20
>> On Dec 21, 2016 17:42, "Eyal Edri" <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>> >
>> >
>> >
>> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>
>> >>
>> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul@redhat.com =
<mailto:ykaul@redhat.com>> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>>>
>> >>>>
>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> =
wrote:
>> >>>>>
>> >>>>>
>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra =
<vfeenstr@redhat.com <mailto:vfeenstr@redhat.com>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren =
<bkorren@redhat.com <mailto:bkorren@redhat.com>> wrote:
>> >>>>>>>>
>> >>>>>>>> The test for running VMs had been failing since yesterday.
>> >>>>>>>>
>> >>>>>>>> The patch merged before the failures started was:
>> >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ =
<https://gerrit.ovirt.org/#/c/68826/>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) =
while running
>> >>>>>>>> this API call:
>> >>>>>>>>
>> >>>>>>>> api.vms.get(VM0_NAME).status.state =3D=3D =E2=80=98up'
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is a REST API call, the patch above is Frontend. So this =
is unrelated.
>> >>>>>>>
>> >>>>>>> However on Host 0 I can see this:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') The vm start process =
failed (vm:615) Traceback (most recent call last): File =
"/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm =
self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run =
self._connection.createXML(domxml, flags), File =
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, =
in wrapper ret =3D f(*args, **kwargs) File =
"/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper =
return func(inst, *args, **kwargs) File =
"/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML =
if ret is None:raise libvirtError('virDomainCreateXML() failed', =
conn=3Dself) libvirtError: internal error: process exited while =
connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: =
CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 =
15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to =
maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z =
qemu-kvm: -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: =
speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) =
to bus "usb.0", port "1" (high speed)=20
>> >>>>>
>> >>>>>
>> >>>>> it is likely related to the recent USB patches=20
>> >>>>> investigating
>> >>>>
>> >>>>
>> >>>> hm, there are multiple problems (features/bugs depending on =
prefferred point of view:)
>> >>>> but there is an easy =E2=80=9Cfix=E2=80=9D taking care of this =
particular problem, so we can start with that and figure out the proper =
approach later
>> >>>> arik will push that and merge it soon, likely today
>> >>>
>> >>>
>> >>> Thanks - if there is a quicker way to resolve this by reverting, =
I think it's a better option.
>> >>
>> >>
>> >> I really need to talk you out of this approach:-)
>> >> It does sound tempting and logical, but with our development model =
of large patch series combined with late detection it really is quite =
risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out the =
right revert patch is more complicated then fixing it.
>> >
>> >
>> > Can we start asking developers run OST before they merge so it will =
be early detection and not late detection?=20
>> > We have video sessions on how to use OST, so it shouldn't be any =
issues in running it on a patch.
>> > =20
>> >>
>> >> I believe the best is to identify it early and notify the =
maintainer who merged that patch ASAP, as that person is in the best =
position to asses if revert is safe or if there is a simple follow up =
patch he can push right away
>> >>
>> >> We can surely improve on reporting, so Barak, how/why did you =
point to that particular patch in your email? It should start failing on =
16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 =
01:56:07 2016 -0500)
>> >> Even though it may be that it was hidden because of =
c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 =
15:16:40 2016 -0500)
>> >>
>> >> (fix is ready, waiting on CI now)
>> >>
>> >> Thanks,
>> >> michal
>> >>
>> >>> Y.
>> >>> =20
>> >>>>
>> >>>>
>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: =
internal error: process exited while connecting to monitor: =
2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any =
NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: speed =
mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus =
"usb.0", port "1" (high speed) (code=3D1) (vm:1197) 2016-12-20 =
16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Stopping connection =
(guestagent:430)=20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> And on The engine loads of these:
>> >>>>>>>
>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command =
'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, =
VdsIdVDSCommandParametersBase:{runAsync=3D'true', =
hostId=3D'994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: =
VDSGenericException: VDSNetworkException: Timeout during rpc call =
2016-12-20 16:53:57,849-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp =
Reactor) [7971dfb4] MESSAGE content-length:80 =
destination:jms.topic.vdsm_responses content-type:application/json =
subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", =
"id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}=EF=BF=BD =
2016-12-20 16:53:57,850-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) =
[] Message received: {"jsonrpc": "2.0", "id": =
"3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 =
16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] =
(ResponseWorker) [] Not able to update response for =
"3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: =
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: =
VDSGenericException: VDSNetworkException: Timeout during rpc call at =
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo=
mmand.java:73) [vdsbroker.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos=
tSetupNetworkPoller.java:56) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet=
upNetworkPoller.java:41) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetu=
pNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at =
org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCom=
mand(HostSetupNetworksCommand.java:287) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas=
e.java:1249) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm=
andBase.java:1389) [bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20=
53) [bll.jar:] at org.ovirt.engine.core.utils.tr =
<http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.execu=
teInSuppressed(TransactionSupport.java:164) [utils.jar:] at =
org.ovirt.engine.core.utils.tr =
<http://org.ovirt.engine.core.utils.tr/>ansaction.TransactionSupport.execu=
teInScope(TransactionSupport.java:103) [utils.jar:] at =
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) =
[bll.jar:] at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) =
[bll.jar:] at =
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(De=
faultBackendActionExecutor.java:13) [bll.jar:] at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] =
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) =
[bll.jar:] at =
org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] =
at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) =
[:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat=
ion(ManagedReferenceMethodInterceptor.java:52) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29=
9BindingsInterceptor.java:70) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29=
9BindingsInterceptor.java:80) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi=
ndingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] =
at =
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv=
ocation(UserInterceptorFactory.java:63) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aro=
undInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at =
sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) =
[:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.proce=
ssInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>vocationmetrics.ExecutionTimeInter=
ceptor.processInvocation(ExecutionTimeInterceptor.java:43) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont=
ext.java:437) at =
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo=
ke(AbstractEJBRequestScopeActivationInterceptor.java:73) =
[weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at =
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati=
on(EjbRequestScopeActivationInterceptor.java:83) =
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(=
ConcurrentContextInterceptor.java:45) =
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept=
or.java:21) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc=
essInvocation(ComponentDispatcherInterceptor.java:52) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.si =
<http://org.jboss.as.ejb3.component.si/>ngleton.SingletonComponentInstance=
AssociationInterceptor.processInvocation(SingletonComponentInstanceAssocia=
tionInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2=
63) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j=
ava:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.CurrentInvocationContex=
tInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41=
) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>vocationmetrics.WaitTimeIntercepto=
r.processInvocation(WaitTimeInterceptor.java:47) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se=
curityContextInterceptor.java:100) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInv=
ocation(StartupAwaitInterceptor.java:22) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.ShutDownInterceptorFact=
ory$1.processInvocation(ShutDownInterceptorFactory.java:64) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ejb3.component.in =
<http://org.jboss.as.ejb3.component.in/>terceptors.LoggingInterceptor.proc=
essInvocation(LoggingInterceptor.java:67) =
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at org.jboss.as.ee.component.Name =
<http://org.jboss.as.ee.component.name/>spaceContextInterceptor.processInv=
ocation(NamespaceContextInterceptor.java:50) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte=
xtClassLoaderInterceptor.java:64) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) =
at =
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur=
ityManager.java:636) at =
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe=
ckingInterceptor.java:61) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) =
at =
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P=
rivilegedWithCombinerInterceptor.java:80) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) =
at =
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript=
ion.java:185) at =
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34=
0) at =
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept=
or.java:61) at =
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan=
dler.java:73) at =
org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unk=
nown Source) [common.jar:] at =
org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendReso=
urce.java:250) at =
org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen=
dResource.java:182) at =
org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen=
dResource.java:170) at =
org.ovirt.engine.api.restapi.resource.BackendHostResource.se =
<http://esource.backendhostresource.se/>tupNetworks(BackendHostResource.ja=
va:212) at =
org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at =
org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.ja=
va:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =
[rt.jar:1.8.0_111] at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:=
62) [rt.jar:1.8.0_111] at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43) [rt.jar:1.8.0_111] at =
java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at =
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:=
139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMetho=
dInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker=
.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour=
ceLocatorInvoker.java:138) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] 	at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok=
er.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour=
ceLocatorInvoker.java:133) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok=
er.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher=
.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher=
.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.servi=
ce(ServletContainerDispatcher.java:221) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht=
tpServletDispatcher.java:56) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht=
tpServletDispatcher.java:51) =
[resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at =
javax.servlet.http.HttpServlet.service(HttpServlet.java:790) =
[jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at =
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.j=
ava:85) at =
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav=
a:81) at =
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq=
uest(ServletSecurityRoleHandler.java:62) at =
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl=
etDispatchingHandler.java:36) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet=
InitialHandler.java:274) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletI=
nitialHandler.java:209) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatch=
erImpl.java:221) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDis=
patcherImpl.java:147) at =
io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherIm=
pl.java:111) at =
org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt=
er.java:139) at =
org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt=
er.java:68) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt=
er.java:116) at =
org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt=
er.java:71) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestAp=
iSessionMgmtFilter.java:78) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFi=
lter.java:39) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(Sso=
RestApiNegotiationFilter.java:91) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApi=
AuthFilter.java:47) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.SessionValidationFilter.do =
<http://ers.sessionvalidationfilter.do/>Filter(SessionValidationFilter.jav=
a:59) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(=
RestApiSessionValidationFilter.java:35) [aaa.jar:] at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr=
otectionFilter.java:111) at =
org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr=
otectionFilter.java:102) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSuppo=
rtFilter.java:183) at =
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) =
at =
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter=
Handler.java:131) at =
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav=
a:84) at =
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq=
uest(ServletSecurityRoleHandler.java:62) at =
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl=
etDispatchingHandler.java:36) at =
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.=
handleRequest(SecurityContextAssociationHandler.java:78) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.han=
dleRequest(SSLInformationAssociationHandler.java:131) at =
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.han=
dleRequest(ServletAuthenticationCallHandler.java:57) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.security.handlers.AuthenticationConstraintHandler.handleReques=
t(AuthenticationConstraintHandler.java:53) at =
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest=
(AbstractConfidentialityHandler.java:46) at =
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHand=
ler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at =
io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.han=
dleRequest(ServletSecurityConstraintHandler.java:59) at =
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleReques=
t(AuthenticationMechanismsHandler.java:60) at =
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.ha=
ndleRequest(CachedAuthenticatedSessionHandler.java:77) at =
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(No=
tificationReceiverHandler.java:50) at =
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.ha=
ndleRequest(AbstractSecurityContextAssociationHandler.java:43) at =
io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRe=
quest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at io.undertow.server.handlers.Pr =
<http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(Predi=
cateHandler.java:43) at =
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(Serv=
letInitialHandler.java:292) at =
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletIniti=
alHandler.java:81) at =
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa=
ndler.java:138) at =
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa=
ndler.java:135) at =
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(Ser=
vletRequestContextThreadSetupAction.java:48) at =
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClass=
LoaderSetupAction.java:43) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread=
SetupActionWrapper.java:44) at =
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet=
InitialHandler.java:272) at =
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletIniti=
alHandler.java:81) at =
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(Servlet=
InitialHandler.java:104) at =
io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at =
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) =
at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142) [rt.jar:1.8.0_111] at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) =
[rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR =
[org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default =
task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM =
response: Internal timeout occured=20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is definitely not related to the patch linked above.
>> >>>>>>>
>> >>>>>>> However I am also not quite sure what is the cause of this =
right now, it might be a breaking change in VDSM.
>> >>>>>>>
>> >>>>>>
>> >>>>>> The DC level is 4.0, is it possible that is the reason for the =
failure?=20
>> >>>>>>
>> >>>>>> =20
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Full test code can be seen here:
>> >>>>>>>> =
https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Dba=
sic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l2=
91 =
<https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Db=
asic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l=
291>
>> >>>>>>>>
>> >>>>>>>> Full test exception can be seen here:
>> >>>>>>>> =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepor=
t/junit/%28root%29/004_basic_sanity/vm_run/ =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepo=
rt/junit/%28root%29/004_basic_sanity/vm_run/>
>> >>>>>>>>
>> >>>>>>>> Further logs can be seen in Jenkins:
>> >>>>>>>> =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/=
exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas=
ic-suite-4.1/post-004_basic_sanity.py/ =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact=
/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba=
sic-suite-4.1/post-004_basic_sanity.py/>
>> >>>>>>>>
>> >>>>>>>> --=20
>> >>>>>>>> Barak Korren
>> >>>>>>>> bkorren@redhat.com <mailto:bkorren@redhat.com>
>> >>>>>>>> RHCE, RHCi, RHV-DevOps Team
>> >>>>>>>> https://ifireball.wordpress.com/ =
<https://ifireball.wordpress.com/>
>> >>>>>>>> _______________________________________________
>> >>>>>>>> Devel mailing list
>> >>>>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> Devel mailing list
>> >>>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --=20
>> >>>>>> Eyal Edri
>> >>>>>> Associate Manager
>> >>>>>> RHV DevOps
>> >>>>>> EMEA ENG Virtualization R&D
>> >>>>>> Red Hat Israel
>> >>>>>>
>> >>>>>> phone: +972-9-7692018 <tel:+972%209-769-2018>
>> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>>>>> _______________________________________________
>> >>>>>> Devel mailing list
>> >>>>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Devel mailing list
>> >>>> Devel@ovirt.org <mailto:Devel@ovirt.org>
>> >>>> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --=20
>> > Eyal Edri
>> > Associate Manager
>> > RHV DevOps
>> > EMEA ENG Virtualization R&D
>> > Red Hat Israel
>> >
>> > phone: +972-9-7692018 <tel:+972%209-769-2018>
>> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >
>> > _______________________________________________
>> > Devel mailing list
>> > Devel@ovirt.org <mailto:Devel@ovirt.org>
>> > http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>=20
>=20
> _______________________________________________
> Devel mailing list
> Devel@ovirt.org <mailto:Devel@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/devel =
<http://lists.ovirt.org/mailman/listinfo/devel>
>=20
> Guys,
> 4.1 is broken for 11 hours now.
> It seems that the issues are accumulating; looking into =
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/=
exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas=
ic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log=
_ovirt-engine/engine.log =
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact=
/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba=
sic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_lo=
g_ovirt-engine/engine.log>
>=20
> shows a duplicate entry in disk mapping:

interesting error, Arik. Reruns should be added to lago
The root cause is few lines above and the same as was fixed yesterday on =
master. It just needs a backport to 4.1=20
> 2016-12-21 10:52:51,436-05 ERROR =
[org.ovirt.engine.core.bll.RunVmOnceCommand] =
(org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: =
java.lang.IllegalStateException: Duplicate key =
org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6=
0b
> 	at =
java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) =
[rt.jar:1.8.0_111]
> 	at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) =
[rt.jar:1.8.0_111]
> 	at =
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1=
374) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47=
1) [rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)=
 [rt.jar:1.8.0_111]
> 	at =
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) =
[rt.jar:1.8.0_111]
> 	at =
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) =
[rt.jar:1.8.0_111]
> 	at =
org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM=
ap(DiskHandler.java:46) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar=
d(RunVmValidator.java:318) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator=
.java:146) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) =
[bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:=
58) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83=
4) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) =
[bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa=
se.java:163) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138=
) [bll.jar:]
> 	at =
org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) =
[bll.jar:]
>=20
> I suggest we consider reverting to the last good state, and work out =
the issues from there.


--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div =
class=3D"">On 22 Dec 2016, at 08:57, Doron Fediuck <<a =
href=3D"mailto:dfediuck@redhat.com" class=3D"">dfediuck@redhat.com</a>>=
 wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div =
dir=3D"ltr" class=3D""><br class=3D""><div class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On Thu, Dec 22, 2016 at 9:48 AM, =
Michal Skrivanek <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>></span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
style=3D"word-wrap:break-word" class=3D""><br class=3D""><div =
class=3D""><span class=3D"gmail-"><blockquote type=3D"cite" =
class=3D""><div class=3D"">On 21 Dec 2016, at 20:52, Eyal Edri <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:</div><br =
class=3D"gmail-m_-6974087903737897314Apple-interchange-newline"><div =
class=3D""><div dir=3D"auto" class=3D"">Not as easy at it sounds,   =
the current flow of ost is much more complicated than just build =
artifacts. I'm not saving we won't do it,  but it won't be ready =
tomorrow and we designed lago exactly for being independent of any ci =
system so anyone can run it on their =
laptop. </div></div></blockquote><div class=3D""><br =
class=3D""></div></span>+1</div><div class=3D"">we just need to improve =
a little bit to make it possible for everyone. Almost there.</div><div =
class=3D""><span class=3D"gmail-"><br class=3D""><blockquote type=3D"cite"=
 class=3D""><div class=3D""><div dir=3D"auto" class=3D""><div dir=3D"auto"=
 class=3D""> So maybe using Jenkins might be simpler but people =
shouldn't skip verification just because such a job doesn't exist =
yet. </div></div><div class=3D"gmail_extra"><br class=3D""><div =
class=3D"gmail_quote">On Dec 21, 2016 9:02 PM, "Oved Ourfali" <<a =
href=3D"mailto:oourfali@redhat.com" target=3D"_blank" =
class=3D"">oourfali@redhat.com</a>> wrote:<br type=3D"attribution" =
class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p =
dir=3D"ltr" class=3D"">Why not run it via Jenkins for patches? <br =
class=3D"">
Like, if you add a comment saying "run: ost" it will run it? <br =
class=3D""></p></blockquote></div></div></div></blockquote><div =
class=3D""><br class=3D""></div></span>please don=E2=80=99t add more. =
Even the Rerun-hooks thing is hard to remember. Even after years I =
always keep asking is it "Re-run hooks=E2=80=9D or =E2=80=9CRerun-Hooks=E2=
=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does case =
matter=E2=80=A6..bleh</div><div class=3D"">Either add the button or =
maybe add it as a link in the automated comments (that would actually =
work good enough)</div><div class=3D""><br class=3D""></div><div =
class=3D"">Thanks,</div><div class=3D"">michal<div class=3D""><div =
class=3D"gmail-h5"><br class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><div class=3D"gmail_extra"><div =
class=3D"gmail_quote"><blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex"><p dir=3D"ltr" class=3D"">
It do it automatically based on another thing? <br class=3D""></p><p =
dir=3D"ltr" class=3D"">On Dec 21, 2016 17:42, "Eyal Edri" <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:<br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
> On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>><br class=3D"">
>><br class=3D"">
>>> On 21 Dec 2016, at 16:25, Yaniv Kaul <<a =
href=3D"mailto:ykaul@redhat.com" target=3D"_blank" =
class=3D"">ykaul@redhat.com</a>> wrote:<br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek <<a =
href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" =
class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <<a =
href=3D"mailto:eedri@redhat.com" target=3D"_blank" =
class=3D"">eedri@redhat.com</a>> wrote:<br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz =
Feenstra <<a href=3D"mailto:vfeenstr@redhat.com" target=3D"_blank" =
class=3D"">vfeenstr@redhat.com</a>> wrote:<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak =
Korren <<a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" =
class=3D"">bkorren@redhat.com</a>> wrote:<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The test for running VMs had been =
failing since yesterday.<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The patch merged before the failures =
started was:<br class=3D"">
>>>>>>>> <a =
href=3D"https://gerrit.ovirt.org/#/c/68826/" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/#/c/6<wbr class=3D"">8826/</a><br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> The error we`re seeing is a time-out =
(after two minutes) while running<br class=3D"">
>>>>>>>> this API call:<br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> api.vms.get(VM0_NAME).status.s<wbr =
class=3D"">tate =3D=3D =E2=80=98up'<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> This is a REST API call, the patch above is =
Frontend. So this is unrelated.<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> However on Host 0 I can see this:<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) =
[virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') =
The vm start process failed (vm:615) Traceback (most recent call last): =
File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm =
self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run =
self._connection.createXML(dom<wbr class=3D"">xml, flags), File =
"/usr/lib/python2.7/site-packa<wbr =
class=3D"">ges/vdsm/libvirtconnection.py"<wbr class=3D"">, line 123, in =
wrapper ret =3D f(*args, **kwargs) File =
"/usr/lib/python2.7/site-packa<wbr class=3D"">ges/vdsm/utils.py", line =
941, in wrapper return func(inst, *args, **kwargs) File =
"/usr/lib64/python2.7/site-pac<wbr class=3D"">kages/libvirt.py", line =
3782, in createXML if ret is None:raise =
libvirtError('virDomainCreateX<wbr class=3D"">ML() failed', conn=3Dself) =
libvirtError: internal error: process exited while connecting to =
monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not =
present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: =
Warning: speed mismatch trying to attach usb device "QEMU USB CCID" =
(full speed) to bus "usb.0", port "1" (high speed) <br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>>> it is likely related to the recent USB =
patches <br class=3D"">
>>>>> investigating<br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>> hm, there are multiple problems (features/bugs =
depending on prefferred point of view:)<br class=3D"">
>>>> but there is an easy =E2=80=9Cfix=E2=80=9D taking care =
of this particular problem, so we can start with that and figure out the =
proper approach later<br class=3D"">
>>>> arik will push that and merge it soon, likely today<br =
class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>>> Thanks - if there is a quicker way to resolve this by =
reverting, I think it's a better option.<br class=3D"">
>><br class=3D"">
>><br class=3D"">
>> I really need to talk you out of this approach:-)<br class=3D"">
>> It does sound tempting and logical, but with our development =
model of large patch series combined with late detection it really is =
quite risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out =
the right revert patch is more complicated then fixing it.<br class=3D"">
><br class=3D"">
><br class=3D"">
> Can we start asking developers run OST before they merge so it will =
be early detection and not late detection? <br class=3D"">
> We have video sessions on how to use OST, so it shouldn't be any =
issues in running it on a patch.<br class=3D"">
>  <br class=3D"">
>><br class=3D"">
>> I believe the best is to identify it early and notify the =
maintainer who merged that patch ASAP, as that person is in the best =
position to asses if revert is safe or if there is a simple follow up =
patch he can push right away<br class=3D"">
>><br class=3D"">
>> We can surely improve on reporting, so Barak, how/why did you =
point to that particular patch in your email? It should start failing on =
16c2ec236184b3152f1df8e874b431<wbr class=3D"">15f78d0989 (CommitDate: =
Fri Dec 16 01:56:07 2016 -0500)<br class=3D"">
>> Even though it may be that it was hidden because =
of c46f653a7846c3c2a76507b8dcf<wbr class=3D"">5bc0391ec5709 =
(CommitDate: Mon Dec 19 15:16:40 2016 -0500)<br class=3D"">
>><br class=3D"">
>> (fix is ready, waiting on CI now)<br class=3D"">
>><br class=3D"">
>> Thanks,<br class=3D"">
>> michal<br class=3D"">
>><br class=3D"">
>>> Y.<br class=3D"">
>>>  <br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) =
[virt.vm] (vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') =
Changed state to Down: internal error: process exited while connecting =
to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not =
present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 =
2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus =
should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: =
-device usb-ccid,id=3Dccid0,bus=3Dusb.0,po<wbr class=3D"">rt=3D1: =
Warning: speed mismatch trying to attach usb device "QEMU USB CCID" =
(full speed) to bus "usb.0", port "1" (high speed) (code=3D1) (vm:1197) =
2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f<wbr class=3D"">-183a6e54def2') Stopping =
connection (guestagent:430) <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> And on The engine loads of these:<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>>> 2016-12-20 16:53:57,844-05 ERROR =
[org.ovirt.engine.core.vdsbrok<wbr class=3D"">er.vdsbroker.PollVDSCommand]=
 (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] =
Command 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, =
VdsIdVDSCommandParametersBase:<wbr class=3D"">{runAsync=3D'true', =
hostId=3D'994b5d79-605f-4415-94f<wbr class=3D"">2-02c79cfa246e'})' =
execution failed: VDSGenericException: VDSNetworkException: Timeout =
during rpc call 2016-12-20 16:53:57,849-05 DEBUG =
[org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.reactors.stomp.impl.Message] (SSL Stomp Reactor) [7971dfb4] =
MESSAGE content-length:80 destination:jms.topic.vdsm_res<wbr =
class=3D"">ponses content-type:application/json =
subscription:5b6494d5-d5a0-477<wbr class=3D"">1-941c-a8be70f72450 =
{"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr =
class=3D"">95c122d", "result": true}=EF=BF=BD 2016-12-20 16:53:57,850-05 =
DEBUG [org.ovirt.vdsm.jsonrpc.client<wbr =
class=3D"">.internal.ResponseWorker] (ResponseWorker) [] Message =
received: {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc73<wbr =
class=3D"">95c122d", "result": true} 2016-12-20 16:53:57,850-05 ERROR =
[org.ovirt.vdsm.jsonrpc.client<wbr class=3D"">.JsonRpcClient] =
(ResponseWorker) [] Not able to update response for =
"3c95fdb0-5b77-4927-9f6e-adc73<wbr class=3D"">95c122d" 2016-12-20 =
16:53:57,844-05 DEBUG [org.ovirt.engine.core.vdsbrok<wbr =
class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) =
[5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Exception: =
org.ovirt.engine.core.vdsbroke<wbr =
class=3D"">r.vdsbroker.VDSNetworkExceptio<wbr class=3D"">n: =
VDSGenericException: VDSNetworkException: Timeout during rpc call 	=
at org.ovirt.engine.core.vdsbroke<wbr =
class=3D"">r.vdsbroker.FutureVDSCommand.<wbr =
class=3D"">get(FutureVDSCommand.java:73) [vdsbroker.jar:] 	at =
org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworkPolle<wbr =
class=3D"">r.getValue(HostSetupNetworkPol<wbr class=3D"">ler.java:56) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworkPolle<wbr =
class=3D"">r.poll(HostSetupNetworkPoller.<wbr class=3D"">java:41) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworksComm<wbr =
class=3D"">and.invokeSetupNetworksCommand<wbr =
class=3D"">(HostSetupNetworksCommand.<wbr class=3D"">java:426) =
[bll.jar:] 	at org.ovirt.engine.core.bll.netw<wbr =
class=3D"">ork.host.HostSetupNetworksComm<wbr =
class=3D"">and.executeCommand(HostSetupNe<wbr =
class=3D"">tworksCommand.java:287) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeWithoutTransact<wbr =
class=3D"">ion(CommandBase.java:1249) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeActionInTransac<wbr =
class=3D"">tionScope(CommandBase.java:<wbr class=3D"">1389) [bll.jar:] 	=
at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.runInTransaction(Comma<wbr class=3D"">ndBase.java:2053)=
 [bll.jar:] 	at <a href=3D"http://org.ovirt.engine.core.utils.tr/" =
class=3D"">org.ovirt.engine.core.utils.tr</a><wbr =
class=3D"">ansaction.TransactionSupport.e<wbr =
class=3D"">xecuteInSuppressed(Transaction<wbr class=3D"">Support.java:164)=
 [utils.jar:] 	at <a href=3D"http://org.ovirt.engine.core.utils.tr/" =
class=3D"">org.ovirt.engine.core.utils.tr</a><wbr =
class=3D"">ansaction.TransactionSupport.e<wbr =
class=3D"">xecuteInScope(TransactionSuppo<wbr class=3D"">rt.java:103) =
[utils.jar:] 	at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.execute(CommandBase.<wbr class=3D"">java:1449) =
[bll.jar:] 	at org.ovirt.engine.core.bll.Comm<wbr =
class=3D"">andBase.executeAction(CommandB<wbr class=3D"">ase.java:395) =
[bll.jar:] 	at org.ovirt.engine.core.bll.exec<wbr =
class=3D"">utor.DefaultBackendActionExecu<wbr =
class=3D"">tor.execute(DefaultBackendActi<wbr =
class=3D"">onExecutor.java:13) [bll.jar:] 	at =
org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runAction(Backend.java:<wbr class=3D"">511) [bll.jar:] 	=
at org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runActionImpl(Backend.java<wbr class=3D"">:493) =
[bll.jar:] 	at org.ovirt.engine.core.bll.Back<wbr =
class=3D"">end.runAction(Backend.java:<wbr class=3D"">446) [bll.jar:] 	=
at sun.reflect.GeneratedMethodAcc<wbr class=3D"">essor232.invoke(Unknown =
Source) [:1.8.0_111] 	at sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.as.ee.component.Mana<wbr =
class=3D"">gedReferenceMethodInterceptor.<wbr =
class=3D"">processInvocation(ManagedRefer<wbr =
class=3D"">enceMethodInterceptor.java:52) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.delegateInte<wbr =
class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:70) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.doMethodInte<wbr =
class=3D"">rception(Jsr299BindingsInterce<wbr class=3D"">ptor.java:80) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.weld.ejb.Jsr299Bi<wbr =
class=3D"">ndingsInterceptor.processInvoc<wbr =
class=3D"">ation(Jsr299BindingsIntercepto<wbr class=3D"">r.java:93) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ee.component.inte<wbr =
class=3D"">rceptors.UserInterceptorFactor<wbr =
class=3D"">y$1.processInvocation(UserInte<wbr =
class=3D"">rceptorFactory.java:63) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.ovirt.engine.core.bll.inte<wbr =
class=3D"">rceptors.CorrelationIdTrackerI<wbr =
class=3D"">nterceptor.aroundInvoke(Correl<wbr =
class=3D"">ationIdTrackerInterceptor.<wbr class=3D"">java:13) [bll.jar:] =
	at sun.reflect.GeneratedMethodAcc<wbr =
class=3D"">essor64.invoke(Unknown Source) [:1.8.0_111] 	at =
sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.as.ee.component.Mana<wbr =
class=3D"">gedReferenceLifecycleMethodInt<wbr =
class=3D"">erceptor.processInvocation(Man<wbr =
class=3D"">agedReferenceLifecycleMethodIn<wbr =
class=3D"">terceptor.java:89) 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">vocationmetrics.ExecutionTimeI<wbr =
class=3D"">nterceptor.processInvocation(E<wbr =
class=3D"">xecutionTimeInterceptor.java:4<wbr class=3D"">3) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext$Invocation.proceed(<wbr =
class=3D"">InterceptorContext.java:437) 	at =
org.jboss.weld.ejb.AbstractEJB<wbr =
class=3D"">RequestScopeActivationIntercep<wbr =
class=3D"">tor.aroundInvoke(AbstractEJBRe<wbr =
class=3D"">questScopeActivationIntercepto<wbr class=3D"">r.java:73) =
[weld-core-impl-2.3.5.Final.ja<wbr class=3D"">r:2.3.5.Final] 	at =
org.jboss.as.weld.ejb.EjbReque<wbr =
class=3D"">stScopeActivationInterceptor.<wbr =
class=3D"">processInvocation(EjbRequestSc<wbr =
class=3D"">opeActivationInterceptor.java:<wbr class=3D"">83) =
[wildfly-weld-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ee.concurrent.Con<wbr =
class=3D"">currentContextInterceptor.proc<wbr =
class=3D"">essInvocation(ConcurrentContex<wbr =
class=3D"">tInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:1<wbr =
class=3D"">0.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.InitialIn<wbr =
class=3D"">terceptor.processInvocation(In<wbr =
class=3D"">itialInterceptor.java:21) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.inte<wbr =
class=3D"">rceptors.ComponentDispatcherIn<wbr =
class=3D"">terceptor.processInvocation(Co<wbr =
class=3D"">mponentDispatcherInterceptor.<wbr class=3D"">java:52) 	=
at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.si/" =
class=3D"">org.jboss.as.ejb3.component.si</a><wbr =
class=3D"">ngleton.SingletonComponentInst<wbr =
class=3D"">anceAssociationInterceptor.pro<wbr =
class=3D"">cessInvocation(SingletonCompon<wbr =
class=3D"">entInstanceAssociationIntercep<wbr class=3D"">tor.java:53) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.invokeInNoTx(CMTTxInte<wbr class=3D"">rceptor.java:263)=
 [wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.supports(CMTTxIntercep<wbr class=3D"">tor.java:374) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.as.ejb3.tx.CMTTxInte<wbr =
class=3D"">rceptor.processInvocation(CMTT<wbr =
class=3D"">xInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.CurrentInvocationCo<wbr =
class=3D"">ntextInterceptor.<wbr =
class=3D"">processInvocation(CurrentInvoc<wbr =
class=3D"">ationContextInterceptor.java:<wbr class=3D"">41) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">vocationmetrics.WaitTimeInterc<wbr =
class=3D"">eptor.processInvocation(WaitTi<wbr =
class=3D"">meInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.security.Sec<wbr =
class=3D"">urityContextInterceptor.proces<wbr =
class=3D"">sInvocation(SecurityContextInt<wbr =
class=3D"">erceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.as.ejb3.deployment.p<wbr =
class=3D"">rocessors.StartupAwaitIntercep<wbr =
class=3D"">tor.processInvocation(StartupA<wbr =
class=3D"">waitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar<wbr =
class=3D"">:10.1.0.Final] 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.ShutDownInterceptor<wbr =
class=3D"">Factory$1.processInvocation(Sh<wbr =
class=3D"">utDownInterceptorFactory.java:<wbr class=3D"">64) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ejb3.component.in/" =
class=3D"">org.jboss.as.ejb3.component.in</a><wbr =
class=3D"">terceptors.LoggingInterceptor.<wbr =
class=3D"">processInvocation(LoggingInter<wbr class=3D"">ceptor.java:67) =
[wildfly-ejb3-10.1.0.Final.jar<wbr class=3D"">:10.1.0.Final] 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at <a href=3D"http://org.jboss.as.ee.component.name/" =
class=3D"">org.jboss.as.ee.component.Name</a><wbr =
class=3D"">spaceContextInterceptor.proces<wbr =
class=3D"">sInvocation(NamespaceContextIn<wbr =
class=3D"">terceptor.java:50) 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ContextCl<wbr =
class=3D"">assLoaderInterceptor.processIn<wbr =
class=3D"">vocation(ContextClassLoaderInt<wbr class=3D"">erceptor.java:64)=
 	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) 	=
at org.wildfly.security.manager.W<wbr =
class=3D"">ildFlySecurityManager.doChecke<wbr =
class=3D"">d(WildFlySecurityManager.java:<wbr class=3D"">636) 	at =
org.jboss.invocation.AccessChe<wbr =
class=3D"">ckingInterceptor.processInvoca<wbr =
class=3D"">tion(AccessCheckingInterceptor<wbr class=3D"">.java:61) 	=
at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.run(InterceptorConte<wbr class=3D"">xt.java:356) 	=
at org.jboss.invocation.Privilege<wbr =
class=3D"">dWithCombinerInterceptor.<wbr =
class=3D"">processInvocation(PrivilegedWi<wbr =
class=3D"">thCombinerInterceptor.java:80) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.View<wbr =
class=3D"">Service$View.invoke(ViewServic<wbr class=3D"">e.java:198) 	=
at org.jboss.as.ee.component.View<wbr =
class=3D"">Description$1.processInvocatio<wbr =
class=3D"">n(ViewDescription.java:185) 	at =
org.jboss.invocation.Intercept<wbr =
class=3D"">orContext.proceed(InterceptorC<wbr class=3D"">ontext.java:340) =
	at org.jboss.invocation.ChainedIn<wbr =
class=3D"">terceptor.processInvocation(Ch<wbr =
class=3D"">ainedInterceptor.java:61) 	at =
org.jboss.as.ee.component.Prox<wbr =
class=3D"">yInvocationHandler.invoke(Prox<wbr =
class=3D"">yInvocationHandler.java:73) 	at =
org.ovirt.engine.core.common.i<wbr =
class=3D"">nterfaces.BackendLocal$$$view4<wbr =
class=3D"">.runAction(Unknown Source) [common.jar:] 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.doActi<wbr =
class=3D"">on(BackendResource.java:250) 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.perfor<wbr =
class=3D"">mAction(BackendResource.java:<wbr class=3D"">182) 	at =
org.ovirt.engine.api.restapi.r<wbr =
class=3D"">esource.BackendResource.perfor<wbr =
class=3D"">mAction(BackendResource.java:<wbr class=3D"">170) 	at =
org.ovirt.engine.api.restapi.r<wbr class=3D""><a =
href=3D"http://esource.backendhostresource.se/" =
class=3D"">esource.BackendHostResource.se</a><wbr =
class=3D"">tupNetworks(BackendHostResourc<wbr class=3D"">e.java:212) 	=
at org.ovirt.engine.api.v3.V3Serv<wbr =
class=3D"">er.adaptAction(V3Server.java:<wbr class=3D"">216) 	at =
org.ovirt.engine.api.v3.server<wbr =
class=3D"">s.V3HostServer.setupNetworks(<wbr =
class=3D"">V3HostServer.java:182) 	at =
sun.reflect.NativeMethodAccess<wbr class=3D"">orImpl.invoke0(Native =
Method) [rt.jar:1.8.0_111] 	at sun.reflect.NativeMethodAccess<wbr =
class=3D"">orImpl.invoke(NativeMethodAcce<wbr class=3D"">ssorImpl.java:62)=
 [rt.jar:1.8.0_111] 	at sun.reflect.DelegatingMethodAc<wbr =
class=3D"">cessorImpl.invoke(DelegatingMe<wbr =
class=3D"">thodAccessorImpl.java:43) [rt.jar:1.8.0_111] 	at =
java.lang.reflect.Method.invok<wbr class=3D"">e(Method.java:498) =
[rt.jar:1.8.0_111] 	at org.jboss.resteasy.core.Method<wbr =
class=3D"">InjectorImpl.invoke(MethodInje<wbr =
class=3D"">ctorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceMethodInvoker.invokeOnTarget<wbr =
class=3D"">(ResourceMethodInvoker.java:29<wbr class=3D"">5) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceMethodInvoker.invoke(Resourc<wbr =
class=3D"">eMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invokeOnTarge<wbr =
class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:138) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invoke(Resour<wbr =
class=3D"">ceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invokeOnTarge<wbr =
class=3D"">tObject(ResourceLocatorInvoker<wbr class=3D"">.java:133) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.core.Resour<wbr =
class=3D"">ceLocatorInvoker.invoke(Resour<wbr =
class=3D"">ceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Synchr<wbr =
class=3D"">onousDispatcher.invoke(Synchro<wbr =
class=3D"">nousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.core.Synchr<wbr =
class=3D"">onousDispatcher.invoke(Synchro<wbr =
class=3D"">nousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.ServletContainerDi<wbr =
class=3D"">spatcher.service(ServletContai<wbr =
class=3D"">nerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.j<wbr =
class=3D"">ar:3.0.19.Final] 	at org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.HttpServletDispatc<wbr =
class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:56) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
org.jboss.resteasy.plugins.ser<wbr =
class=3D"">ver.servlet.HttpServletDispatc<wbr =
class=3D"">her.service(HttpServletDispatc<wbr class=3D"">her.java:51) =
[resteasy-jaxrs-3.0.19.Final.j<wbr class=3D"">ar:3.0.19.Final] 	at =
javax.servlet.http.HttpServlet<wbr =
class=3D"">.service(HttpServlet.java:790) =
[jboss-servlet-api_3.1_spec-1.<wbr class=3D"">0.0.Final.jar:1.0.0.Final] =
	at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletHandler.handleRequest(Se<wbr =
class=3D"">rvletHandler.java:85) 	at =
io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler.handleRequest(Fil<wbr =
class=3D"">terHandler.java:81) 	at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityRoleHan<wbr =
class=3D"">dler.handleRequest(ServletSecu<wbr =
class=3D"">rityRoleHandler.java:62) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletDispatchingHandler.handl<wbr =
class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:274) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchT<wbr =
class=3D"">oPath(ServletInitialHandler.<wbr class=3D"">java:209) 	=
at io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forwardImpl(R<wbr =
class=3D"">equestDispatcherImpl.java:221) 	at =
io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forwardImplSe<wbr =
class=3D"">tup(RequestDispatcherImpl.<wbr class=3D"">java:147) 	at =
io.undertow.servlet.spec.Reque<wbr =
class=3D"">stDispatcherImpl.forward(Reque<wbr =
class=3D"">stDispatcherImpl.java:111) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.VersionFilter.doFilt<wbr =
class=3D"">er(VersionFilter.java:139) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.VersionFilter.doFilt<wbr =
class=3D"">er(VersionFilter.java:68) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.CurrentFilter.doFilt<wbr =
class=3D"">er(CurrentFilter.java:116) 	at =
org.ovirt.engine.api.restapi.i<wbr =
class=3D"">nvocation.CurrentFilter.doFilt<wbr =
class=3D"">er(CurrentFilter.java:71) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.RestApiSessionMgmtFilter.d<wbr =
class=3D"">oFilter(RestApiSessionMgmtFilt<wbr class=3D"">er.java:78) =
[aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.EnforceAuthFilter.doFilter<wbr =
class=3D"">(EnforceAuthFilter.java:39) [aaa.jar:] 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.SsoRestApiNegotiationFilte<wbr =
class=3D"">r.doFilter(SsoRestApiNegotiati<wbr class=3D"">onFilter.java:91)=
 [aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.SsoRestApiAuthFilter.doFil<wbr =
class=3D"">ter(SsoRestApiAuthFilter.java:<wbr class=3D"">47) [aaa.jar:] 	=
at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr class=3D""><a =
href=3D"http://ers.sessionvalidationfilter.do/" =
class=3D"">ers.SessionValidationFilter.do</a><wbr =
class=3D"">Filter(SessionValidationFilter<wbr class=3D"">.java:59) =
[aaa.jar:] 	at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.core.aaa.filt<wbr =
class=3D"">ers.RestApiSessionValidationFi<wbr =
class=3D"">lter.doFilter(RestApiSessionVa<wbr =
class=3D"">lidationFilter.java:35) [aaa.jar:] 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CSRFProtectionFilter.d<wbr =
class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:111) 	=
at org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CSRFProtectionFilter.d<wbr =
class=3D"">oFilter(CSRFProtectionFilter.j<wbr class=3D"">ava:102) 	=
at io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
org.ovirt.engine.api.restapi.s<wbr =
class=3D"">ecurity.CORSSupportFilter.doFi<wbr =
class=3D"">lter(CORSSupportFilter.java:<wbr class=3D"">183) 	at =
io.undertow.servlet.core.Manag<wbr =
class=3D"">edFilter.doFilter(ManagedFilte<wbr class=3D"">r.java:61) 	=
at io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler$FilterChainImpl.d<wbr =
class=3D"">oFilter(FilterHandler.java:131<wbr class=3D"">) 	at =
io.undertow.servlet.handlers.F<wbr =
class=3D"">ilterHandler.handleRequest(Fil<wbr =
class=3D"">terHandler.java:84) 	at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityRoleHan<wbr =
class=3D"">dler.handleRequest(ServletSecu<wbr =
class=3D"">rityRoleHandler.java:62) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletDispatchingHandler.handl<wbr =
class=3D"">eRequest(ServletDispatchingHan<wbr class=3D"">dler.java:36) 	=
at org.wildfly.extension.undertow<wbr =
class=3D"">.security.SecurityContextAssoc<wbr =
class=3D"">iationHandler.handleRequest(Se<wbr =
class=3D"">curityContextAssociationHandle<wbr class=3D"">r.java:78) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.SSLInformationAssociat<wbr =
class=3D"">ionHandler.handleRequest(SSLIn<wbr =
class=3D"">formationAssociationHandler.<wbr class=3D"">java:131) 	=
at io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletAuthenticationC<wbr =
class=3D"">allHandler.handleRequest(Servl<wbr =
class=3D"">etAuthenticationCallHandler.<wbr class=3D"">java:57) 	=
at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.security.handlers.<wbr =
class=3D"">AuthenticationConstraintHandle<wbr =
class=3D"">r.handleRequest(Authentication<wbr =
class=3D"">ConstraintHandler.java:53) 	at =
io.undertow.security.handlers.<wbr =
class=3D"">AbstractConfidentialityHandler<wbr =
class=3D"">.handleRequest(AbstractConfide<wbr =
class=3D"">ntialityHandler.java:46) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletConfidentiality<wbr =
class=3D"">ConstraintHandler.<wbr =
class=3D"">handleRequest(ServletConfident<wbr =
class=3D"">ialityConstraintHandler.java:<wbr class=3D"">64) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.ServletSecurityConstra<wbr =
class=3D"">intHandler.handleRequest(Servl<wbr =
class=3D"">etSecurityConstraintHandler.<wbr class=3D"">java:59) 	=
at io.undertow.security.handlers.<wbr =
class=3D"">AuthenticationMechanismsHandle<wbr =
class=3D"">r.handleRequest(Authentication<wbr =
class=3D"">MechanismsHandler.java:60) 	at =
io.undertow.servlet.handlers.s<wbr =
class=3D"">ecurity.CachedAuthenticatedSes<wbr =
class=3D"">sionHandler.handleRequest(Cach<wbr =
class=3D"">edAuthenticatedSessionHandler.<wbr class=3D"">java:77) 	=
at io.undertow.security.handlers.<wbr =
class=3D"">NotificationReceiverHandler.ha<wbr =
class=3D"">ndleRequest(NotificationReceiv<wbr =
class=3D"">erHandler.java:50) 	at io.undertow.security.handlers.<wbr =
class=3D"">AbstractSecurityContextAssocia<wbr =
class=3D"">tionHandler.handleRequest(Abst<wbr =
class=3D"">ractSecurityContextAssociation<wbr class=3D"">Handler.java:43) =
	at <a href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
org.wildfly.extension.undertow<wbr =
class=3D"">.security.jacc.JACCContextIdHa<wbr =
class=3D"">ndler.handleRequest(JACCContex<wbr =
class=3D"">tIdHandler.java:61) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at <a =
href=3D"http://io.undertow.server.handlers.pr/" =
class=3D"">io.undertow.server.handlers.Pr</a><wbr =
class=3D"">edicateHandler.handleRequest(P<wbr =
class=3D"">redicateHandler.java:43) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.handleFir<wbr =
class=3D"">stRequest(ServletInitialHandle<wbr class=3D"">r.java:292) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.access$10<wbr =
class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$2.call(Se<wbr =
class=3D"">rvletInitialHandler.java:138) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$2.call(Se<wbr =
class=3D"">rvletInitialHandler.java:135) 	at =
io.undertow.servlet.core.Servl<wbr =
class=3D"">etRequestContextThreadSetupAct<wbr =
class=3D"">ion$1.call(ServletRequestConte<wbr =
class=3D"">xtThreadSetupAction.java:48) 	at =
io.undertow.servlet.core.Conte<wbr =
class=3D"">xtClassLoaderSetupAction$1.<wbr =
class=3D"">call(ContextClassLoaderSetupAc<wbr class=3D"">tion.java:43) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.api.Legacy<wbr =
class=3D"">ThreadSetupActionWrapper$1.<wbr =
class=3D"">call(LegacyThreadSetupActionWr<wbr class=3D"">apper.java:44) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.dispatchR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:272) 	=
at io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler.access$00<wbr =
class=3D"">0(ServletInitialHandler.java:<wbr class=3D"">81) 	at =
io.undertow.servlet.handlers.S<wbr =
class=3D"">ervletInitialHandler$1.handleR<wbr =
class=3D"">equest(ServletInitialHandler.<wbr class=3D"">java:104) 	=
at io.undertow.server.Connectors.<wbr =
class=3D"">executeRootHandler(Connectors.<wbr class=3D"">java:202) 	=
at io.undertow.server.HttpServerE<wbr =
class=3D"">xchange$1.run(HttpServerExchan<wbr class=3D"">ge.java:805) 	=
at java.util.concurrent.ThreadPoo<wbr =
class=3D"">lExecutor.runWorker(ThreadPool<wbr =
class=3D"">Executor.java:1142) [rt.jar:1.8.0_111] 	at =
java.util.concurrent.ThreadPoo<wbr =
class=3D"">lExecutor$Worker.run(ThreadPoo<wbr =
class=3D"">lExecutor.java:617) [rt.jar:1.8.0_111] 	at =
java.lang.Thread.run(Thread.ja<wbr class=3D"">va:745) [rt.jar:1.8.0_111] =
2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.vdsbrok<wbr =
class=3D"">er.vdsbroker.PollVDSCommand] (default task-17) =
[5ecd5a55-2b7a-4dd6-b42b-cc49b<wbr class=3D"">bfb3962] Timeout waiting =
for VDSM response: Internal timeout occured <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> This is definitely not related to the patch =
linked above.<br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> However I am also not quite sure what is =
the cause of this right now, it might be a breaking change in VDSM.<br =
class=3D"">
>>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> The DC level is 4.0, is it possible that is the =
reason for the failure? <br class=3D"">
>>>>>><br class=3D"">
>>>>>>  <br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Full test code can be seen here:<br =
class=3D"">
>>>>>>>> <a =
href=3D"https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblo=
b;f=3Dbasic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/m=
aster#l291" target=3D"_blank" =
class=3D"">https://gerrit.ovirt.org/gitwe<wbr =
class=3D"">b?p=3Dovirt-system-tests.git;a=3D<wbr =
class=3D"">blob;f=3Dbasic-suite-4.1/test-<wbr =
class=3D"">scenarios/004_basic_sanity.py;<wbr =
class=3D"">hb=3Drefs/heads/master#l291</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Full test exception can be seen =
here:<br class=3D"">
>>>>>>>> <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/t=
estReport/junit/%28root%29/004_basic_sanity/vm_run/" target=3D"_blank" =
class=3D"">http://jenkins.ovirt.org/job/t<wbr =
class=3D"">est-repo_ovirt_experimental_4.<wbr =
class=3D"">1/13/testReport/junit/%28root%<wbr =
class=3D"">29/004_basic_sanity/vm_run/</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> Further logs can be seen in Jenkins:<br =
class=3D"">
>>>>>>>> <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/a=
rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_=
logs/basic-suite-4.1/post-004_basic_sanity.py/" target=3D"_blank" =
class=3D"">http://jenkins.ovirt.org/job/t<wbr =
class=3D"">est-repo_ovirt_experimental_4.<wbr =
class=3D"">1/13/artifact/exported-artifac<wbr =
class=3D"">ts/basic_suite_4.1.sh-el7/<wbr =
class=3D"">exported-artifacts/test_logs/<wbr =
class=3D"">basic-suite-4.1/post-004_basic<wbr =
class=3D"">_sanity.py/</a><br class=3D"">
>>>>>>>><br class=3D"">
>>>>>>>> -- <br class=3D"">
>>>>>>>> Barak Korren<br class=3D"">
>>>>>>>> <a href=3D"mailto:bkorren@redhat.com" =
target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D"">
>>>>>>>> RHCE, RHCi, RHV-DevOps Team<br =
class=3D"">
>>>>>>>> <a =
href=3D"https://ifireball.wordpress.com/" target=3D"_blank" =
class=3D"">https://ifireball.wordpress.co<wbr class=3D"">m/</a><br =
class=3D"">
>>>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>>>> Devel mailing list<br class=3D"">
>>>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>><br class=3D"">
>>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>>> Devel mailing list<br class=3D"">
>>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>><br class=3D"">
>>>>>> -- <br class=3D"">
>>>>>> Eyal Edri<br class=3D"">
>>>>>> Associate Manager<br class=3D"">
>>>>>> RHV DevOps<br class=3D"">
>>>>>> EMEA ENG Virtualization R&D<br class=3D"">
>>>>>> Red Hat Israel<br class=3D"">
>>>>>><br class=3D"">
>>>>>> phone: <a href=3D"tel:+972%209-769-2018" =
value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br =
class=3D"">
>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br =
class=3D"">
>>>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>>>> Devel mailing list<br class=3D"">
>>>>>> <a href=3D"mailto:Devel@ovirt.org" =
target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>>>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>>>><br class=3D"">
>>>>><br class=3D"">
>>>><br class=3D"">
>>>><br class=3D"">
>>>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>>>> Devel mailing list<br class=3D"">
>>>> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
>>>> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel"=
 target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D"">
>>><br class=3D"">
>>><br class=3D"">
>><br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
> -- <br class=3D"">
> Eyal Edri<br class=3D"">
> Associate Manager<br class=3D"">
> RHV DevOps<br class=3D"">
> EMEA ENG Virtualization R&D<br class=3D"">
> Red Hat Israel<br class=3D"">
><br class=3D"">
> phone: <a href=3D"tel:+972%209-769-2018" value=3D"+97297692018" =
target=3D"_blank" class=3D"">+972-9-7692018</a><br class=3D"">
> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br class=3D"">
><br class=3D"">
> ______________________________<wbr class=3D"">_________________<br =
class=3D"">
> Devel mailing list<br class=3D"">
> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" =
class=3D"">Devel@ovirt.org</a><br class=3D"">
> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/devel</a><br class=3D""></p>
</blockquote></div></div>
</div></blockquote></div></div></div><br class=3D""></div><br =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">
Devel mailing list<br class=3D"">
<a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br =
class=3D"">
<a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br=
 class=3D""></div><div class=3D"gmail_extra">Guys,</div><div =
class=3D"gmail_extra">4.1 is broken for 11 hours now.</div><div =
class=3D"gmail_extra">It seems that the issues are accumulating; looking =
into <a =
href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/a=
rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_=
logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/=
_var_log_ovirt-engine/engine.log" =
class=3D"">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/2=
8/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/te=
st_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engi=
ne/_var_log_ovirt-engine/engine.log</a></div><div =
class=3D"gmail_extra"><br class=3D""></div><div =
class=3D"gmail_extra">shows a duplicate entry in disk =
mapping:</div></div></div></blockquote><div><br =
class=3D""></div>interesting error, Arik. Reruns should be added to =
lago</div><div>The root cause is few lines above and the same as was =
fixed yesterday on master. It just needs a backport to 4.1 <br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"gmail_extra"><pre style=3D"" =
class=3D"">2016-12-21 10:52:51,436-05 ERROR =
[org.ovirt.engine.core.bll.RunVmOnceCommand] =
(org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: =
java.lang.IllegalStateException: Duplicate key =
org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f6=
0b
	at =
java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) =
[rt.jar:1.8.0_111]
	at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111]
	at =
java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) =
[rt.jar:1.8.0_111]
	at =
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1=
374) [rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:47=
1) [rt.jar:1.8.0_111]
	at =
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)=
 [rt.jar:1.8.0_111]
	at =
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) =
[rt.jar:1.8.0_111]
	at =
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) =
[rt.jar:1.8.0_111]
	at =
org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementM=
ap(DiskHandler.java:46) [bll.jar:]
	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscar=
d(RunVmValidator.java:318) [bll.jar:]
	at =
org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator=
.java:146) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) =
[bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:=
58) [bll.jar:]
	at =
org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:83=
4) [bll.jar:]
	at =
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) =
[bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBa=
se.java:163) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138=
) [bll.jar:]
	at =
org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) =
[bll.jar:]</pre><pre style=3D"" class=3D""><br class=3D""></pre><pre =
style=3D"" class=3D"">I suggest we consider reverting to the last good =
state, and work out the issues from there.</pre></div></div>
</div></blockquote></div><br class=3D""></body></html>=

--Apple-Mail=_A7242C61-4E1B-4E36-A61C-1428000C13F3--

Re: [ovirt-devel] System tests for 4.1 currently failing to run VMs!

Oved Ourfali

Eyal Edri

Michal Skrivanek

Oved Ourfali

Doron Fediuck

Michal Skrivanek

Michal Skrivanek

Fred Rolland

Doron Fediuck

Fred Rolland

Michal Skrivanek

Michal Skrivanek

tags

participants (6)