--Apple-Mail=_4A67E93F-035B-42BA-9727-09586A1C7969 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 21 Dec 2016, at 20:52, Eyal Edri <eedri@redhat.com> wrote: =20 Not as easy at it sounds, the current flow of ost is much more = complicated than just build artifacts. I'm not saving we won't do it, = but it won't be ready tomorrow and we designed lago exactly for being = independent of any ci system so anyone can run it on their laptop.=20
+1 we just need to improve a little bit to make it possible for everyone. = Almost there.
So maybe using Jenkins might be simpler but people shouldn't skip = verification just because such a job doesn't exist yet.=20 =20 On Dec 21, 2016 9:02 PM, "Oved Ourfali" <oourfali@redhat.com = <mailto:oourfali@redhat.com>> wrote: Why not run it via Jenkins for patches?=20 Like, if you add a comment saying "run: ost" it will run it?=20 =20
It do it automatically based on another thing?=20 =20 On Dec 21, 2016 17:42, "Eyal Edri" <eedri@redhat.com = <mailto:eedri@redhat.com>> wrote:
On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote:
On 21 Dec 2016, at 16:25, Yaniv Kaul <ykaul@redhat.com =
<mailto:ykaul@redhat.com>> wrote:
On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote:
On 21 Dec 2016, at 14:56, Michal Skrivanek =
<michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> = wrote:
> On 21 Dec 2016, at 12:19, Eyal Edri <eedri@redhat.com =
<mailto:eedri@redhat.com>> wrote:
> > > > On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra = <vfeenstr@redhat.com <mailto:vfeenstr@redhat.com>> wrote: >> >> >>> On Dec 21, 2016, at 11:17 AM, Barak Korren = <bkorren@redhat.com <mailto:bkorren@redhat.com>> wrote: >>> >>> The test for running VMs had been failing since yesterday. >>> >>> The patch merged before the failures started was: >>> https://gerrit.ovirt.org/#/c/68826/ = <https://gerrit.ovirt.org/#/c/68826/> >> >> >> >> >>> >>> The error we`re seeing is a time-out (after two minutes) = while running >>> this API call: >>> >>> api.vms.get(VM0_NAME).status.state =3D=3D =E2=80=98up' >> >> >> This is a REST API call, the patch above is Frontend. So this = is unrelated. >> >> However on Host 0 I can see this: >> >> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] = (vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') The vm start process = failed (vm:615) Traceback (most recent call last): File = "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm = self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run = self._connection.createXML(domxml, flags), File = "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, = in wrapper ret =3D f(*args, **kwargs) File = "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper = return func(inst, *args, **kwargs) File = "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in createXML = if ret is None:raise libvirtError('virDomainCreateXML() failed', = conn=3Dself) libvirtError: internal error: process exited while = connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: = CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 = 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to = maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z = qemu-kvm: -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: = speed mismatch trying to attach usb device "QEMU USB CCID" (full speed) = to bus "usb.0", port "1" (high speed)=20
it is likely related to the recent USB patches=20 investigating
hm, there are multiple problems (features/bugs depending on =
but there is an easy =E2=80=9Cfix=E2=80=9D taking care of this =
arik will push that and merge it soon, likely today
Thanks - if there is a quicker way to resolve this by reverting, I =
I really need to talk you out of this approach:-) It does sound tempting and logical, but with our development model =
of large patch series combined with late detection it really is quite = risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out the = right revert patch is more complicated then fixing it.
Can we start asking developers run OST before they merge so it will = be early detection and not late detection?=20 We have video sessions on how to use OST, so it shouldn't be any = issues in running it on a patch. =20
I believe the best is to identify it early and notify the =
We can surely improve on reporting, so Barak, how/why did you point =
to that particular patch in your email? It should start failing on = 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 = 01:56:07 2016 -0500)
Even though it may be that it was hidden because of = c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 = 15:16:40 2016 -0500)
(fix is ready, waiting on CI now)
Thanks, michal
Y. =20
>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] =
(vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: = internal error: process exited while connecting to monitor: = 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any = NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 = 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus = should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: = -device usb-ccid,id=3Dccid0,bus=3Dusb.0,port=3D1: Warning: speed = mismatch trying to attach usb device "QEMU USB CCID" (full speed) to bus = "usb.0", port "1" (high speed) (code=3D1) (vm:1197) 2016-12-20 = 16:54:43,550 INFO (vm/d299ab29) [virt.vm] = (vmId=3D'd299ab29-284a-435c-a50f-183a6e54def2') Stopping connection = (guestagent:430)=20
>> >> >> >> And on The engine loads of these: >> >> 2016-12-20 16:53:57,844-05 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command = 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: = VDSGenericException: VDSNetworkException: Timeout during rpc call = 2016-12-20 16:53:57,849-05 DEBUG = [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp = Reactor) [7971dfb4] MESSAGE content-length:80 = destination:jms.topic.vdsm_responses content-type:application/json = subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", = "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}=EF=BF=BD = 2016-12-20 16:53:57,850-05 DEBUG = [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) = [] Message received: {"jsonrpc": "2.0", "id": = "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 = 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] = (ResponseWorker) [] Not able to update response for = "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 DEBUG = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: = VDSGenericException: VDSNetworkException: Timeout during rpc call at = org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCo= mmand.java:73) [vdsbroker.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(Hos= tSetupNetworkPoller.java:56) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSet= upNetworkPoller.java:41) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetu=
please don=E2=80=99t add more. Even the Rerun-hooks thing is hard to = remember. Even after years I always keep asking is it "Re-run hooks=E2=80=9D= or =E2=80=9CRerun-Hooks=E2=80=9D or =E2=80=9Crerun hooks=E2=80=9D, does = case matter=E2=80=A6..bleh Either add the button or maybe add it as a link in the automated = comments (that would actually work good enough) Thanks, michal prefferred point of view:) particular problem, so we can start with that and figure out the proper = approach later think it's a better option. maintainer who merged that patch ASAP, as that person is in the best = position to asses if revert is safe or if there is a simple follow up = patch he can push right away pNetworksCommand(HostSetupNetworksCommand.java:426) [bll.jar:] at = org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCom= mand(HostSetupNetworksCommand.java:287) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBas= e.java:1249) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(Comm= andBase.java:1389) [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:20= 53) [bll.jar:] at = org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppre= ssed(TransactionSupport.java:164) [utils.jar:] at = org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(= TransactionSupport.java:103) [utils.jar:] at = org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) = [bll.jar:] at = org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) = [bll.jar:] at = org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(De= faultBackendActionExecutor.java:13) [bll.jar:] at = org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] = at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) = [bll.jar:] at = org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] = at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) = [:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocat= ion(ManagedReferenceMethodInterceptor.java:52) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr29= 9BindingsInterceptor.java:70) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr29= 9BindingsInterceptor.java:80) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299Bi= ndingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] = at = org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInv= ocation(UserInterceptorFactory.java:63) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aro= undInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at = sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) = [:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.proce= ssInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.pro= cessInvocation(ExecutionTimeInterceptor.java:43) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCont= ext.java:437) at = org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvo= ke(AbstractEJBRequestScopeActivationInterceptor.java:73) = [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at = org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocati= on(EjbRequestScopeActivationInterceptor.java:83) = [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(= ConcurrentContextInterceptor.java:45) = [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InitialInterceptor.processInvocation(InitialIntercept= or.java:21) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.proc= essInvocation(ComponentDispatcherInterceptor.java:52) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociatio= nInterceptor.processInvocation(SingletonComponentInstanceAssociationInterc= eptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:2= 63) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.j= ava:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextIntercept= or.processInvocation(CurrentInvocationContextInterceptor.java:41) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processI= nvocation(WaitTimeInterceptor.java:47) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(Se= curityContextInterceptor.java:100) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInv= ocation(StartupAwaitInterceptor.java:22) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.proc= essInvocation(ShutDownInterceptorFactory.java:64) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocat= ion(LoggingInterceptor.java:67) = [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(Na= mespaceContextInterceptor.java:50) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(Conte= xtClassLoaderInterceptor.java:64) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) = at = org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecur= ityManager.java:636) at = org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessChe= ckingInterceptor.java:61) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) = at = org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(P= rivilegedWithCombinerInterceptor.java:80) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) = at = org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescript= ion.java:185) at = org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:34= 0) at = org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedIntercept= or.java:61) at = org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHan= dler.java:73) at = org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unk= nown Source) [common.jar:] at = org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendReso= urce.java:250) at = org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen= dResource.java:182) at = org.ovirt.engine.api.restapi.resource.BackendResource.performAction(Backen= dResource.java:170) at = org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(Ba= ckendHostResource.java:212) at = org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at = org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.ja= va:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) = [rt.jar:1.8.0_111] at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 62) [rt.jar:1.8.0_111] at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) [rt.jar:1.8.0_111] at = java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:= 139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMetho= dInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker= .java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour= ceLocatorInvoker.java:138) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok= er.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(Resour= ceLocatorInvoker.java:133) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvok= er.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher= .java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher= .java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.servi= ce(ServletContainerDispatcher.java:221) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht= tpServletDispatcher.java:56) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ht= tpServletDispatcher.java:51) = [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at = javax.servlet.http.HttpServlet.service(HttpServlet.java:790) = [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at = io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.j= ava:85) at = io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav= a:81) at = io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq= uest(ServletSecurityRoleHandler.java:62) at = io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl= etDispatchingHandler.java:36) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet= InitialHandler.java:274) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletI= nitialHandler.java:209) at = io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatch= erImpl.java:221) at = io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDis= patcherImpl.java:147) at = io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherIm= pl.java:111) at = org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt= er.java:139) at = org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilt= er.java:68) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt= er.java:116) at = org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilt= er.java:71) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestAp= iSessionMgmtFilter.java:78) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFi= lter.java:39) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(Sso= RestApiNegotiationFilter.java:91) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApi= AuthFilter.java:47) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(Session= ValidationFilter.java:59) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(= RestApiSessionValidationFilter.java:35) [aaa.jar:] at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr= otectionFilter.java:111) at = org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFPr= otectionFilter.java:102) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSuppo= rtFilter.java:183) at = io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) = at = io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Filter= Handler.java:131) at = io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.jav= a:84) at = io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleReq= uest(ServletSecurityRoleHandler.java:62) at = io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Servl= etDispatchingHandler.java:36) at = org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.= handleRequest(SecurityContextAssociationHandler.java:78) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.han= dleRequest(SSLInformationAssociationHandler.java:131) at = io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.han= dleRequest(ServletAuthenticationCallHandler.java:57) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.security.handlers.AuthenticationConstraintHandler.handleReques= t(AuthenticationConstraintHandler.java:53) at = io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest= (AbstractConfidentialityHandler.java:46) at = io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHand= ler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at = io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.han= dleRequest(ServletSecurityConstraintHandler.java:59) at = io.undertow.security.handlers.AuthenticationMechanismsHandler.handleReques= t(AuthenticationMechanismsHandler.java:60) at = io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.ha= ndleRequest(CachedAuthenticatedSessionHandler.java:77) at = io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(No= tificationReceiverHandler.java:50) at = io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.ha= ndleRequest(AbstractSecurityContextAssociationHandler.java:43) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRe= quest(JACCContextIdHandler.java:61) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandle= r.java:43) at = io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(Serv= letInitialHandler.java:292) at = io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletIniti= alHandler.java:81) at = io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa= ndler.java:138) at = io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHa= ndler.java:135) at = io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(Ser= vletRequestContextThreadSetupAction.java:48) at = io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClass= LoaderSetupAction.java:43) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThread= SetupActionWrapper.java:44) at = io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Servlet= InitialHandler.java:272) at = io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletIniti= alHandler.java:81) at = io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(Servlet= InitialHandler.java:104) at = io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at = io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) = at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1142) [rt.jar:1.8.0_111] at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) = [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM = response: Internal timeout occured=20
>> >> >> This is definitely not related to the patch linked above. >> >> However I am also not quite sure what is the cause of this = right now, it might be a breaking change in VDSM. >> > > The DC level is 4.0, is it possible that is the reason for the = failure?=20 > > =20 >> >> >>> >>> Full test code can be seen here: >>> = https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Dba= sic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l2= 91 = <https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblob;f=3Db= asic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/master#l= 291> >>> >>> Full test exception can be seen here: >>> = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepor= t/junit/%28root%29/004_basic_sanity/vm_run/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testRepo= rt/junit/%28root%29/004_basic_sanity/vm_run/> >>> >>> Further logs can be seen in Jenkins: >>> = http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/= exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/bas= ic-suite-4.1/post-004_basic_sanity.py/ = <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact= /exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/ba= sic-suite-4.1/post-004_basic_sanity.py/> >>> >>> --=20 >>> Barak Korren >>> bkorren@redhat.com <mailto:bkorren@redhat.com> >>> RHCE, RHCi, RHV-DevOps Team >>> https://ifireball.wordpress.com/ = <https://ifireball.wordpress.com/> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org <mailto:Devel@ovirt.org> >>> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> >> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org <mailto:Devel@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> > > > > > --=20 > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 <tel:+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) > _______________________________________________ > Devel mailing list > Devel@ovirt.org <mailto:Devel@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
--=20 Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 <tel:+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
--Apple-Mail=_4A67E93F-035B-42BA-9727-09586A1C7969 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 21 Dec 2016, at 20:52, Eyal Edri <<a = href=3D"mailto:eedri@redhat.com" class=3D"">eedri@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"auto" class=3D"">Not as easy at it sounds, the current = flow of ost is much more complicated than just build artifacts. I'm not = saving we won't do it, but it won't be ready tomorrow and we = designed lago exactly for being independent of any ci system so anyone = can run it on their laptop. </div></div></blockquote><div><br = class=3D""></div>+1</div><div>we just need to improve a little bit to = make it possible for everyone. Almost there.</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"auto" class=3D""><div dir=3D"auto" class=3D""> So maybe = using Jenkins might be simpler but people shouldn't skip verification = just because such a job doesn't exist yet. </div></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On Dec = 21, 2016 9:02 PM, "Oved Ourfali" <<a = href=3D"mailto:oourfali@redhat.com" class=3D"">oourfali@redhat.com</a>>= wrote:<br type=3D"attribution" class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><p dir=3D"ltr" class=3D"">Why not run it via = Jenkins for patches? <br class=3D""> Like, if you add a comment saying "run: ost" it will run it? <br = class=3D""></p></blockquote></div></div></div></blockquote><div><br = class=3D""></div>please don=E2=80=99t add more. Even the Rerun-hooks = thing is hard to remember. Even after years I always keep asking is it = "Re-run hooks=E2=80=9D or =E2=80=9CRerun-Hooks=E2=80=9D or =E2=80=9Crerun = hooks=E2=80=9D, does case matter=E2=80=A6..bleh</div><div>Either add the = button or maybe add it as a link in the automated comments (that would = actually work good enough)</div><div><br = class=3D""></div><div>Thanks,</div><div>michal<br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"ltr" = class=3D""> It do it automatically based on another thing? <br class=3D""></p><p = dir=3D"ltr" class=3D"">On Dec 21, 2016 17:42, "Eyal Edri" <<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D"">eedri@redhat.com</a>> wrote:<br class=3D""> ><br class=3D""> ><br class=3D""> ><br class=3D""> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek <<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> >><br class=3D""> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <<a = href=3D"mailto:ykaul@redhat.com" target=3D"_blank" = class=3D"">ykaul@redhat.com</a>> wrote:<br class=3D""> >>><br class=3D""> >>><br class=3D""> >>><br class=3D""> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek <<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek <<a = href=3D"mailto:michal.skrivanek@redhat.com" target=3D"_blank" = class=3D"">michal.skrivanek@redhat.com</a>> wrote:<br class=3D""> >>>>><br class=3D""> >>>>><br class=3D""> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D"">eedri@redhat.com</a>> wrote:<br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz = Feenstra <<a href=3D"mailto:vfeenstr@redhat.com" target=3D"_blank" = class=3D"">vfeenstr@redhat.com</a>> wrote:<br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak = Korren <<a href=3D"mailto:bkorren@redhat.com" target=3D"_blank" = class=3D"">bkorren@redhat.com</a>> wrote:<br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> The test for running VMs had been = failing since yesterday.<br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> The patch merged before the failures = started was:<br class=3D""> >>>>>>>> <a = href=3D"https://gerrit.ovirt.org/#/c/68826/" target=3D"_blank" = class=3D"">https://gerrit.ovirt.org/#/c/<wbr class=3D"">68826/</a><br = class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> The error we`re seeing is a time-out = (after two minutes) while running<br class=3D""> >>>>>>>> this API call:<br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> api.vms.get(VM0_NAME).status.<wbr = class=3D"">state =3D=3D =E2=80=98up'<br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>> This is a REST API call, the patch above is = Frontend. So this is unrelated.<br class=3D""> >>>>>>><br class=3D""> >>>>>>> However on Host 0 I can see this:<br = class=3D""> >>>>>>><br class=3D""> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) = [virt.vm] (vmId=3D'd299ab29-284a-435c-<wbr class=3D"">a50f-183a6e54def2') = The vm start process failed (vm:615) Traceback (most recent call last): = File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm = self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run = self._connection.createXML(<wbr class=3D"">domxml, flags), File = "/usr/lib/python2.7/site-<wbr class=3D"">packages/vdsm/<wbr = class=3D"">libvirtconnection.py", line 123, in wrapper ret =3D f(*args, = **kwargs) File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/utils.py", line 941, in wrapper return = func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-<wbr = class=3D"">packages/libvirt.py", line 3782, in createXML if ret is = None:raise libvirtError('<wbr class=3D"">virDomainCreateXML() failed', = conn=3Dself) libvirtError: internal error: process exited while = connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: = CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 = 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to = maxcpus should be described in NUMA config 2016-12-20T21:54:43.101886Z = qemu-kvm: -device usb-ccid,id=3Dccid0,bus=3Dusb.0,<wbr class=3D"">port=3D1= : Warning: speed mismatch trying to attach usb device "QEMU USB CCID" = (full speed) to bus "usb.0", port "1" (high speed) <br class=3D""> >>>>><br class=3D""> >>>>><br class=3D""> >>>>> it is likely related to the recent USB = patches <br class=3D""> >>>>> investigating<br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> hm, there are multiple problems (features/bugs = depending on prefferred point of view:)<br class=3D""> >>>> but there is an easy =E2=80=9Cfix=E2=80=9D taking care = of this particular problem, so we can start with that and figure out the = proper approach later<br class=3D""> >>>> arik will push that and merge it soon, likely today<br = class=3D""> >>><br class=3D""> >>><br class=3D""> >>> Thanks - if there is a quicker way to resolve this by = reverting, I think it's a better option.<br class=3D""> >><br class=3D""> >><br class=3D""> >> I really need to talk you out of this approach:-)<br class=3D""> >> It does sound tempting and logical, but with our development = model of large patch series combined with late detection it really is = quite risky. Here it wouldn=E2=80=99t help much=E2=80=A6and figuring out = the right revert patch is more complicated then fixing it.<br class=3D""> ><br class=3D""> ><br class=3D""> > Can we start asking developers run OST before they merge so it will = be early detection and not late detection? <br class=3D""> > We have video sessions on how to use OST, so it shouldn't be any = issues in running it on a patch.<br class=3D""> > <br class=3D""> >><br class=3D""> >> I believe the best is to identify it early and notify the = maintainer who merged that patch ASAP, as that person is in the best = position to asses if revert is safe or if there is a simple follow up = patch he can push right away<br class=3D""> >><br class=3D""> >> We can surely improve on reporting, so Barak, how/why did you = point to that particular patch in your email? It should start failing on = 16c2ec236184b3152f1df8e874b431<wbr class=3D"">15f78d0989 (CommitDate: = Fri Dec 16 01:56:07 2016 -0500)<br class=3D""> >> Even though it may be that it was hidden because of <wbr = class=3D"">c46f653a7846c3c2a76507b8dcf5bc<wbr class=3D"">0391ec5709 = (CommitDate: Mon Dec 19 15:16:40 2016 -0500)<br class=3D""> >><br class=3D""> >> (fix is ready, waiting on CI now)<br class=3D""> >><br class=3D""> >> Thanks,<br class=3D""> >> michal<br class=3D""> >><br class=3D""> >>> Y.<br class=3D""> >>> <br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) = [virt.vm] (vmId=3D'd299ab29-284a-435c-<wbr class=3D"">a50f-183a6e54def2') = Changed state to Down: internal error: process exited while connecting = to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not = present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 = 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus = should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: = -device usb-ccid,id=3Dccid0,bus=3Dusb.0,<wbr class=3D"">port=3D1: = Warning: speed mismatch trying to attach usb device "QEMU USB CCID" = (full speed) to bus "usb.0", port "1" (high speed) (code=3D1) (vm:1197) = 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] = (vmId=3D'd299ab29-284a-435c-<wbr class=3D"">a50f-183a6e54def2') Stopping = connection (guestagent:430) <br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>> And on The engine loads of these:<br = class=3D""> >>>>>>><br class=3D""> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR = [org.ovirt.engine.core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr = class=3D"">PollVDSCommand] (default task-17) = [5ecd5a55-2b7a-4dd6-b42b-<wbr class=3D"">cc49bbfb3962] Command = 'PollVDSCommand(HostName =3D lago-basic-suite-4-1-host0, = VdsIdVDSCommandParametersBase:<wbr class=3D"">{runAsync=3D'true', = hostId=3D'994b5d79-605f-4415-<wbr class=3D"">94f2-02c79cfa246e'})' = execution failed: VDSGenericException: VDSNetworkException: Timeout = during rpc call 2016-12-20 16:53:57,849-05 DEBUG = [org.ovirt.vdsm.jsonrpc.<wbr class=3D"">client.reactors.stomp.impl.<wbr = class=3D"">Message] (SSL Stomp Reactor) [7971dfb4] MESSAGE = content-length:80 destination:jms.topic.vdsm_<wbr class=3D"">responses = content-type:application/json subscription:5b6494d5-d5a0-<wbr = class=3D"">4771-941c-a8be70f72450 {"jsonrpc": "2.0", "id": = "3c95fdb0-5b77-4927-9f6e-<wbr class=3D"">adc7395c122d", "result": = true}=EF=BF=BD 2016-12-20 16:53:57,850-05 DEBUG = [org.ovirt.vdsm.jsonrpc.<wbr class=3D"">client.internal.<wbr = class=3D"">ResponseWorker] (ResponseWorker) [] Message received: = {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-<wbr = class=3D"">adc7395c122d", "result": true} 2016-12-20 16:53:57,850-05 = ERROR [org.ovirt.vdsm.jsonrpc.<wbr class=3D"">client.JsonRpcClient] = (ResponseWorker) [] Not able to update response for = "3c95fdb0-5b77-4927-9f6e-<wbr class=3D"">adc7395c122d" 2016-12-20 = 16:53:57,844-05 DEBUG [org.ovirt.engine.core.<wbr = class=3D"">vdsbroker.vdsbroker.<wbr class=3D"">PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-<wbr class=3D"">cc49bbfb3962] = Exception: org.ovirt.engine.core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr = class=3D"">VDSNetworkException: VDSGenericException: = VDSNetworkException: Timeout during rpc call at = org.ovirt.engine.core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr = class=3D"">FutureVDSCommand.get(<wbr class=3D"">FutureVDSCommand.java:73) = [vdsbroker.jar:] at org.ovirt.engine.core.bll.<wbr = class=3D"">network.host.<wbr class=3D"">HostSetupNetworkPoller.<wbr = class=3D"">getValue(<wbr class=3D"">HostSetupNetworkPoller.java:<wbr = class=3D"">56) [bll.jar:] at org.ovirt.engine.core.bll.<wbr = class=3D"">network.host.<wbr class=3D"">HostSetupNetworkPoller.poll(<wbr = class=3D"">HostSetupNetworkPoller.java:<wbr class=3D"">41) [bll.jar:] = at org.ovirt.engine.core.bll.<wbr class=3D"">network.host.<wbr = class=3D"">HostSetupNetworksCommand.<wbr = class=3D"">invokeSetupNetworksCommand(<wbr = class=3D"">HostSetupNetworksCommand.java:<wbr class=3D"">426) [bll.jar:] = at org.ovirt.engine.core.bll.<wbr class=3D"">network.host.<wbr = class=3D"">HostSetupNetworksCommand.<wbr class=3D"">executeCommand(<wbr = class=3D"">HostSetupNetworksCommand.java:<wbr class=3D"">287) [bll.jar:] = at org.ovirt.engine.core.bll.<wbr class=3D"">CommandBase.<wbr = class=3D"">executeWithoutTransaction(<wbr = class=3D"">CommandBase.java:1249) [bll.jar:] at = org.ovirt.engine.core.bll.<wbr class=3D"">CommandBase.<wbr = class=3D"">executeActionInTransactionScop<wbr = class=3D"">e(CommandBase.java:1389) [bll.jar:] at = org.ovirt.engine.core.bll.<wbr = class=3D"">CommandBase.runInTransaction(<wbr = class=3D"">CommandBase.java:2053) [bll.jar:] at = org.ovirt.engine.core.utils.<wbr class=3D"">transaction.<wbr = class=3D"">TransactionSupport.<wbr class=3D"">executeInSuppressed(<wbr = class=3D"">TransactionSupport.java:164) [utils.jar:] at = org.ovirt.engine.core.utils.<wbr class=3D"">transaction.<wbr = class=3D"">TransactionSupport.<wbr class=3D"">executeInScope(<wbr = class=3D"">TransactionSupport.java:103) [utils.jar:] at = org.ovirt.engine.core.bll.<wbr class=3D"">CommandBase.execute(<wbr = class=3D"">CommandBase.java:1449) [bll.jar:] at = org.ovirt.engine.core.bll.<wbr class=3D"">CommandBase.executeAction(<wbr = class=3D"">CommandBase.java:395) [bll.jar:] at = org.ovirt.engine.core.bll.<wbr class=3D"">executor.<wbr = class=3D"">DefaultBackendActionExecutor.<wbr class=3D"">execute(<wbr = class=3D"">DefaultBackendActionExecutor.<wbr class=3D"">java:13) = [bll.jar:] at org.ovirt.engine.core.bll.<wbr = class=3D"">Backend.runAction(Backend.<wbr class=3D"">java:511) = [bll.jar:] at org.ovirt.engine.core.bll.<wbr = class=3D"">Backend.runActionImpl(Backend.<wbr class=3D"">java:493) = [bll.jar:] at org.ovirt.engine.core.bll.<wbr = class=3D"">Backend.runAction(Backend.<wbr class=3D"">java:446) = [bll.jar:] at sun.reflect.<wbr = class=3D"">GeneratedMethodAccessor232.<wbr class=3D"">invoke(Unknown = Source) [:1.8.0_111] at sun.reflect.<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">invoke(<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">java:43) = [rt.jar:1.8.0_111] at java.lang.reflect.Method.<wbr = class=3D"">invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.<wbr = class=3D"">ManagedReferenceMethodIntercep<wbr = class=3D"">tor.processInvocation(<wbr = class=3D"">ManagedReferenceMethodIntercep<wbr class=3D"">tor.java:52) = at org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext$Invocation.<wbr = class=3D"">proceed(InterceptorContext.<wbr class=3D"">java:437) = at org.jboss.as.weld.ejb.<wbr class=3D"">Jsr299BindingsInterceptor.<wbr = class=3D"">delegateInterception(<wbr = class=3D"">Jsr299BindingsInterceptor.<wbr class=3D"">java:70) = [wildfly-weld-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.as.weld.ejb.<wbr class=3D"">Jsr299BindingsInterceptor.<wbr = class=3D"">doMethodInterception(<wbr = class=3D"">Jsr299BindingsInterceptor.<wbr class=3D"">java:80) = [wildfly-weld-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.as.weld.ejb.<wbr class=3D"">Jsr299BindingsInterceptor.<wbr = class=3D"">processInvocation(<wbr = class=3D"">Jsr299BindingsInterceptor.<wbr class=3D"">java:93) = [wildfly-weld-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.as.ee.component.<wbr class=3D"">interceptors.<wbr = class=3D"">UserInterceptorFactory$1.<wbr class=3D"">processInvocation(<wbr= class=3D"">UserInterceptorFactory.java:<wbr class=3D"">63) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext$Invocation.<wbr = class=3D"">proceed(InterceptorContext.<wbr class=3D"">java:437) = at org.ovirt.engine.core.bll.<wbr class=3D"">interceptors.<wbr = class=3D"">CorrelationIdTrackerIntercepto<wbr = class=3D"">r.aroundInvoke(<wbr = class=3D"">CorrelationIdTrackerIntercepto<wbr class=3D"">r.java:13) = [bll.jar:] at sun.reflect.<wbr = class=3D"">GeneratedMethodAccessor64.<wbr class=3D"">invoke(Unknown = Source) [:1.8.0_111] at sun.reflect.<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">invoke(<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">java:43) = [rt.jar:1.8.0_111] at java.lang.reflect.Method.<wbr = class=3D"">invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.as.ee.component.<wbr = class=3D"">ManagedReferenceLifecycleMetho<wbr class=3D"">dInterceptor.<wbr= class=3D"">processInvocation(<wbr = class=3D"">ManagedReferenceLifecycleMetho<wbr = class=3D"">dInterceptor.java:89) at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">invocationmetrics.<wbr = class=3D"">ExecutionTimeInterceptor.<wbr class=3D"">processInvocation(<wbr= class=3D"">ExecutionTimeInterceptor.java:<wbr class=3D"">43) = [wildfly-ejb3-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext$Invocation.<wbr = class=3D"">proceed(InterceptorContext.<wbr class=3D"">java:437) = at org.jboss.weld.ejb.<wbr class=3D"">AbstractEJBRequestScopeActivat<wbr = class=3D"">ionInterceptor.aroundInvoke(<wbr = class=3D"">AbstractEJBRequestScopeActivat<wbr = class=3D"">ionInterceptor.java:73) [weld-core-impl-2.3.5.Final.<wbr = class=3D"">jar:2.3.5.Final] at org.jboss.as.weld.ejb.<wbr = class=3D"">EjbRequestScopeActivationInter<wbr = class=3D"">ceptor.processInvocation(<wbr = class=3D"">EjbRequestScopeActivationInter<wbr class=3D"">ceptor.java:83) = [wildfly-weld-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ee.concurrent.<wbr = class=3D"">ConcurrentContextInterceptor.<wbr = class=3D"">processInvocation(<wbr = class=3D"">ConcurrentContextInterceptor.<wbr class=3D"">java:45) = [wildfly-ee-10.1.0.Final.jar:<wbr class=3D"">10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InitialInterceptor.<wbr = class=3D"">processInvocation(<wbr class=3D"">InitialInterceptor.java:21) = at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">ChainedInterceptor.<wbr = class=3D"">processInvocation(<wbr class=3D"">ChainedInterceptor.java:61) = at org.jboss.as.ee.component.<wbr class=3D"">interceptors.<wbr = class=3D"">ComponentDispatcherInterceptor<wbr = class=3D"">.processInvocation(<wbr = class=3D"">ComponentDispatcherInterceptor<wbr class=3D"">.java:52) = at org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">singleton.<wbr = class=3D"">SingletonComponentInstanceAsso<wbr = class=3D"">ciationInterceptor.<wbr class=3D"">processInvocation(<wbr = class=3D"">SingletonComponentInstanceAsso<wbr = class=3D"">ciationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.tx.<wbr class=3D"">CMTTxInterceptor.invokeInNoTx(<wbr = class=3D"">CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.as.ejb3.tx.<wbr = class=3D"">CMTTxInterceptor.supports(<wbr = class=3D"">CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.as.ejb3.tx.<wbr = class=3D"">CMTTxInterceptor.<wbr class=3D"">processInvocation(<wbr = class=3D"">CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">interceptors.<wbr = class=3D"">CurrentInvocationContextInterc<wbr = class=3D"">eptor.processInvocation(<wbr = class=3D"">CurrentInvocationContextInterc<wbr class=3D"">eptor.java:41) = [wildfly-ejb3-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">invocationmetrics.<wbr = class=3D"">WaitTimeInterceptor.<wbr class=3D"">processInvocation(<wbr = class=3D"">WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.security.<wbr class=3D"">SecurityContextInterceptor.<wbr= class=3D"">processInvocation(<wbr = class=3D"">SecurityContextInterceptor.<wbr class=3D"">java:100) = [wildfly-ejb3-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.deployment.<wbr class=3D"">processors.<wbr = class=3D"">StartupAwaitInterceptor.<wbr class=3D"">processInvocation(<wbr = class=3D"">StartupAwaitInterceptor.java:<wbr class=3D"">22) = [wildfly-ejb3-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">interceptors.<wbr = class=3D"">ShutDownInterceptorFactory$1.<wbr = class=3D"">processInvocation(<wbr = class=3D"">ShutDownInterceptorFactory.<wbr class=3D"">java:64) = [wildfly-ejb3-10.1.0.Final.<wbr class=3D"">jar:10.1.0.Final] at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ejb3.component.<wbr class=3D"">interceptors.<wbr = class=3D"">LoggingInterceptor.<wbr class=3D"">processInvocation(<wbr = class=3D"">LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.<wbr = class=3D"">jar:10.1.0.Final] at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.as.ee.component.<wbr class=3D"">NamespaceContextInterceptor.<wbr= class=3D"">processInvocation(<wbr = class=3D"">NamespaceContextInterceptor.<wbr class=3D"">java:50) = at org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">ContextClassLoaderInterceptor.<wbr = class=3D"">processInvocation(<wbr = class=3D"">ContextClassLoaderInterceptor.<wbr class=3D"">java:64) = at org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.run(<wbr = class=3D"">InterceptorContext.java:356) at = org.wildfly.security.manager.<wbr class=3D"">WildFlySecurityManager.<wbr = class=3D"">doChecked(<wbr class=3D"">WildFlySecurityManager.java:<wbr = class=3D"">636) at org.jboss.invocation.<wbr = class=3D"">AccessCheckingInterceptor.<wbr = class=3D"">processInvocation(<wbr = class=3D"">AccessCheckingInterceptor.<wbr class=3D"">java:61) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">InterceptorContext.run(<wbr = class=3D"">InterceptorContext.java:356) at = org.jboss.invocation.<wbr class=3D"">PrivilegedWithCombinerIntercep<wbr = class=3D"">tor.processInvocation(<wbr = class=3D"">PrivilegedWithCombinerIntercep<wbr class=3D"">tor.java:80) = at org.jboss.invocation.<wbr class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">ChainedInterceptor.<wbr = class=3D"">processInvocation(<wbr class=3D"">ChainedInterceptor.java:61) = at org.jboss.as.ee.component.<wbr = class=3D"">ViewService$View.invoke(<wbr class=3D"">ViewService.java:198) = at org.jboss.as.ee.component.<wbr = class=3D"">ViewDescription$1.<wbr class=3D"">processInvocation(<wbr = class=3D"">ViewDescription.java:185) at org.jboss.invocation.<wbr = class=3D"">InterceptorContext.proceed(<wbr = class=3D"">InterceptorContext.java:340) at = org.jboss.invocation.<wbr class=3D"">ChainedInterceptor.<wbr = class=3D"">processInvocation(<wbr class=3D"">ChainedInterceptor.java:61) = at org.jboss.as.ee.component.<wbr = class=3D"">ProxyInvocationHandler.invoke(<wbr = class=3D"">ProxyInvocationHandler.java:<wbr class=3D"">73) at = org.ovirt.engine.core.common.<wbr = class=3D"">interfaces.BackendLocal$$$<wbr = class=3D"">view4.runAction(Unknown Source) [common.jar:] at = org.ovirt.engine.api.restapi.<wbr class=3D"">resource.BackendResource.<wbr= class=3D"">doAction(BackendResource.java:<wbr class=3D"">250) at = org.ovirt.engine.api.restapi.<wbr class=3D"">resource.BackendResource.<wbr= class=3D"">performAction(BackendResource.<wbr class=3D"">java:182) = at org.ovirt.engine.api.restapi.<wbr = class=3D"">resource.BackendResource.<wbr = class=3D"">performAction(BackendResource.<wbr class=3D"">java:170) = at org.ovirt.engine.api.restapi.<wbr = class=3D"">resource.BackendHostResource.<wbr class=3D"">setupNetworks(<wbr= class=3D"">BackendHostResource.java:212) at = org.ovirt.engine.api.v3.<wbr class=3D"">V3Server.adaptAction(V3Server.<wbr= class=3D"">java:216) at org.ovirt.engine.api.v3.<wbr = class=3D"">servers.V3HostServer.<wbr = class=3D"">setupNetworks(V3HostServer.<wbr class=3D"">java:182) = at sun.reflect.<wbr class=3D"">NativeMethodAccessorImpl.<wbr = class=3D"">invoke0(Native Method) [rt.jar:1.8.0_111] at = sun.reflect.<wbr class=3D"">NativeMethodAccessorImpl.<wbr = class=3D"">invoke(<wbr class=3D"">NativeMethodAccessorImpl.java:<wbr = class=3D"">62) [rt.jar:1.8.0_111] at sun.reflect.<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">invoke(<wbr = class=3D"">DelegatingMethodAccessorImpl.<wbr class=3D"">java:43) = [rt.jar:1.8.0_111] at java.lang.reflect.Method.<wbr = class=3D"">invoke(Method.java:498) [rt.jar:1.8.0_111] at = org.jboss.resteasy.core.<wbr class=3D"">MethodInjectorImpl.invoke(<wbr = class=3D"">MethodInjectorImpl.java:139) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">ResourceMethodInvoker.<wbr = class=3D"">invokeOnTarget(<wbr class=3D"">ResourceMethodInvoker.java:<wbr = class=3D"">295) [resteasy-jaxrs-3.0.19.Final.<wbr = class=3D"">jar:3.0.19.Final] at org.jboss.resteasy.core.<wbr = class=3D"">ResourceMethodInvoker.invoke(<wbr = class=3D"">ResourceMethodInvoker.java:<wbr class=3D"">249) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">ResourceLocatorInvoker.<wbr = class=3D"">invokeOnTargetObject(<wbr = class=3D"">ResourceLocatorInvoker.java:<wbr class=3D"">138) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">ResourceLocatorInvoker.invoke(<wbr= class=3D"">ResourceLocatorInvoker.java:<wbr class=3D"">107) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">ResourceLocatorInvoker.<wbr = class=3D"">invokeOnTargetObject(<wbr = class=3D"">ResourceLocatorInvoker.java:<wbr class=3D"">133) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">ResourceLocatorInvoker.invoke(<wbr= class=3D"">ResourceLocatorInvoker.java:<wbr class=3D"">101) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">SynchronousDispatcher.invoke(<wbr = class=3D"">SynchronousDispatcher.java:<wbr class=3D"">402) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.core.<wbr class=3D"">SynchronousDispatcher.invoke(<wbr = class=3D"">SynchronousDispatcher.java:<wbr class=3D"">209) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.plugins.<wbr class=3D"">server.servlet.<wbr = class=3D"">ServletContainerDispatcher.<wbr class=3D"">service(<wbr = class=3D"">ServletContainerDispatcher.<wbr class=3D"">java:221) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.plugins.<wbr class=3D"">server.servlet.<wbr = class=3D"">HttpServletDispatcher.service(<wbr = class=3D"">HttpServletDispatcher.java:56) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = org.jboss.resteasy.plugins.<wbr class=3D"">server.servlet.<wbr = class=3D"">HttpServletDispatcher.service(<wbr = class=3D"">HttpServletDispatcher.java:51) = [resteasy-jaxrs-3.0.19.Final.<wbr class=3D"">jar:3.0.19.Final] at = javax.servlet.http.<wbr class=3D"">HttpServlet.service(<wbr = class=3D"">HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.<wbr = class=3D"">0.0.Final.jar:1.0.0.Final] at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletHandler.handleRequest(<wbr = class=3D"">ServletHandler.java:85) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler.handleRequest(<wbr = class=3D"">FilterHandler.java:81) at = io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">ServletSecurityRoleHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">ServletSecurityRoleHandler.<wbr class=3D"">java:62) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletDispatchingHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">ServletDispatchingHandler.<wbr class=3D"">java:36) at = io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.servlet.handlers.<wbr class=3D"">ServletInitialHandler.<wbr= class=3D"">dispatchRequest(<wbr = class=3D"">ServletInitialHandler.java:<wbr class=3D"">274) at = io.undertow.servlet.handlers.<wbr class=3D"">ServletInitialHandler.<wbr = class=3D"">dispatchToPath(<wbr class=3D"">ServletInitialHandler.java:<wbr = class=3D"">209) at io.undertow.servlet.spec.<wbr = class=3D"">RequestDispatcherImpl.<wbr class=3D"">forwardImpl(<wbr = class=3D"">RequestDispatcherImpl.java:<wbr class=3D"">221) at = io.undertow.servlet.spec.<wbr class=3D"">RequestDispatcherImpl.<wbr = class=3D"">forwardImplSetup(<wbr = class=3D"">RequestDispatcherImpl.java:<wbr class=3D"">147) at = io.undertow.servlet.spec.<wbr = class=3D"">RequestDispatcherImpl.forward(<wbr = class=3D"">RequestDispatcherImpl.java:<wbr class=3D"">111) at = org.ovirt.engine.api.restapi.<wbr class=3D"">invocation.VersionFilter.<wbr= class=3D"">doFilter(VersionFilter.java:<wbr class=3D"">139) at = org.ovirt.engine.api.restapi.<wbr class=3D"">invocation.VersionFilter.<wbr= class=3D"">doFilter(VersionFilter.java:<wbr class=3D"">68) at = io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.api.restapi.<wbr class=3D"">invocation.CurrentFilter.<wbr= class=3D"">doFilter(CurrentFilter.java:<wbr class=3D"">116) at = org.ovirt.engine.api.restapi.<wbr class=3D"">invocation.CurrentFilter.<wbr= class=3D"">doFilter(CurrentFilter.java:<wbr class=3D"">71) at = io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr class=3D"">filters.<wbr = class=3D"">RestApiSessionMgmtFilter.<wbr class=3D"">doFilter(<wbr = class=3D"">RestApiSessionMgmtFilter.java:<wbr class=3D"">78) [aaa.jar:] = at io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr class=3D"">filters.EnforceAuthFilter.<wbr = class=3D"">doFilter(EnforceAuthFilter.<wbr class=3D"">java:39) = [aaa.jar:] at io.undertow.servlet.core.<wbr = class=3D"">ManagedFilter.doFilter(<wbr class=3D"">ManagedFilter.java:61) = at io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr class=3D"">filters.<wbr = class=3D"">SsoRestApiNegotiationFilter.<wbr class=3D"">doFilter(<wbr = class=3D"">SsoRestApiNegotiationFilter.<wbr class=3D"">java:91) = [aaa.jar:] at io.undertow.servlet.core.<wbr = class=3D"">ManagedFilter.doFilter(<wbr class=3D"">ManagedFilter.java:61) = at io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr = class=3D"">filters.SsoRestApiAuthFilter.<wbr = class=3D"">doFilter(SsoRestApiAuthFilter.<wbr class=3D"">java:47) = [aaa.jar:] at io.undertow.servlet.core.<wbr = class=3D"">ManagedFilter.doFilter(<wbr class=3D"">ManagedFilter.java:61) = at io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr class=3D"">filters.<wbr = class=3D"">SessionValidationFilter.<wbr class=3D"">doFilter(<wbr = class=3D"">SessionValidationFilter.java:<wbr class=3D"">59) [aaa.jar:] = at io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.core.aaa.<wbr class=3D"">filters.<wbr = class=3D"">RestApiSessionValidationFilter<wbr class=3D"">.doFilter(<wbr = class=3D"">RestApiSessionValidationFilter<wbr class=3D"">.java:35) = [aaa.jar:] at io.undertow.servlet.core.<wbr = class=3D"">ManagedFilter.doFilter(<wbr class=3D"">ManagedFilter.java:61) = at io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.api.restapi.<wbr = class=3D"">security.CSRFProtectionFilter.<wbr = class=3D"">doFilter(CSRFProtectionFilter.<wbr class=3D"">java:111) = at org.ovirt.engine.api.restapi.<wbr = class=3D"">security.CSRFProtectionFilter.<wbr = class=3D"">doFilter(CSRFProtectionFilter.<wbr class=3D"">java:102) = at io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = org.ovirt.engine.api.restapi.<wbr = class=3D"">security.CORSSupportFilter.<wbr = class=3D"">doFilter(CORSSupportFilter.<wbr class=3D"">java:183) = at io.undertow.servlet.core.<wbr class=3D"">ManagedFilter.doFilter(<wbr = class=3D"">ManagedFilter.java:61) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler$FilterChainImpl.<wbr = class=3D"">doFilter(FilterHandler.java:<wbr class=3D"">131) at = io.undertow.servlet.handlers.<wbr = class=3D"">FilterHandler.handleRequest(<wbr = class=3D"">FilterHandler.java:84) at = io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">ServletSecurityRoleHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">ServletSecurityRoleHandler.<wbr class=3D"">java:62) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletDispatchingHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">ServletDispatchingHandler.<wbr class=3D"">java:36) at = org.wildfly.extension.<wbr class=3D"">undertow.security.<wbr = class=3D"">SecurityContextAssociationHand<wbr = class=3D"">ler.handleRequest(<wbr = class=3D"">SecurityContextAssociationHand<wbr class=3D"">ler.java:78) = at io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">SSLInformationAssociationHandl<wbr = class=3D"">er.handleRequest(<wbr = class=3D"">SSLInformationAssociationHandl<wbr class=3D"">er.java:131) = at io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">ServletAuthenticationCallHandl<wbr = class=3D"">er.handleRequest(<wbr = class=3D"">ServletAuthenticationCallHandl<wbr class=3D"">er.java:57) = at io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.security.handlers.<wbr = class=3D"">AuthenticationConstraintHandle<wbr = class=3D"">r.handleRequest(<wbr = class=3D"">AuthenticationConstraintHandle<wbr class=3D"">r.java:53) = at io.undertow.security.handlers.<wbr = class=3D"">AbstractConfidentialityHandler<wbr = class=3D"">.handleRequest(<wbr = class=3D"">AbstractConfidentialityHandler<wbr class=3D"">.java:46) = at io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">ServletConfidentialityConstrai<wbr = class=3D"">ntHandler.handleRequest(<wbr = class=3D"">ServletConfidentialityConstrai<wbr = class=3D"">ntHandler.java:64) at io.undertow.servlet.handlers.<wbr = class=3D"">security.<wbr class=3D"">ServletSecurityConstraintHandl<wbr = class=3D"">er.handleRequest(<wbr = class=3D"">ServletSecurityConstraintHandl<wbr class=3D"">er.java:59) = at io.undertow.security.handlers.<wbr = class=3D"">AuthenticationMechanismsHandle<wbr = class=3D"">r.handleRequest(<wbr = class=3D"">AuthenticationMechanismsHandle<wbr class=3D"">r.java:60) = at io.undertow.servlet.handlers.<wbr class=3D"">security.<wbr = class=3D"">CachedAuthenticatedSessionHand<wbr = class=3D"">ler.handleRequest(<wbr = class=3D"">CachedAuthenticatedSessionHand<wbr class=3D"">ler.java:77) = at io.undertow.security.handlers.<wbr = class=3D"">NotificationReceiverHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">NotificationReceiverHandler.<wbr class=3D"">java:50) = at io.undertow.security.handlers.<wbr = class=3D"">AbstractSecurityContextAssocia<wbr = class=3D"">tionHandler.handleRequest(<wbr = class=3D"">AbstractSecurityContextAssocia<wbr = class=3D"">tionHandler.java:43) at = io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at org.wildfly.extension.<wbr class=3D"">undertow.security.jacc.<wbr = class=3D"">JACCContextIdHandler.<wbr class=3D"">handleRequest(<wbr = class=3D"">JACCContextIdHandler.java:61) at = io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.server.handlers.<wbr class=3D"">PredicateHandler.<wbr = class=3D"">handleRequest(<wbr class=3D"">PredicateHandler.java:43) = at io.undertow.servlet.handlers.<wbr class=3D"">ServletInitialHandler.<wbr= class=3D"">handleFirstRequest(<wbr = class=3D"">ServletInitialHandler.java:<wbr class=3D"">292) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletInitialHandler.access$<wbr = class=3D"">100(ServletInitialHandler.<wbr class=3D"">java:81) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletInitialHandler$2.call(<wbr = class=3D"">ServletInitialHandler.java:<wbr class=3D"">138) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletInitialHandler$2.call(<wbr = class=3D"">ServletInitialHandler.java:<wbr class=3D"">135) at = io.undertow.servlet.core.<wbr = class=3D"">ServletRequestContextThreadSet<wbr = class=3D"">upAction$1.call(<wbr = class=3D"">ServletRequestContextThreadSet<wbr class=3D"">upAction.java:48)= at io.undertow.servlet.core.<wbr = class=3D"">ContextClassLoaderSetupAction$<wbr class=3D"">1.call(<wbr = class=3D"">ContextClassLoaderSetupAction.<wbr class=3D"">java:43) = at io.undertow.servlet.api.<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">$1.call(<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">.java:44) = at io.undertow.servlet.api.<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">$1.call(<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">.java:44) = at io.undertow.servlet.api.<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">$1.call(<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">.java:44) = at io.undertow.servlet.api.<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">$1.call(<wbr = class=3D"">LegacyThreadSetupActionWrapper<wbr class=3D"">.java:44) = at io.undertow.servlet.handlers.<wbr class=3D"">ServletInitialHandler.<wbr= class=3D"">dispatchRequest(<wbr = class=3D"">ServletInitialHandler.java:<wbr class=3D"">272) at = io.undertow.servlet.handlers.<wbr = class=3D"">ServletInitialHandler.access$<wbr = class=3D"">000(ServletInitialHandler.<wbr class=3D"">java:81) at = io.undertow.servlet.handlers.<wbr class=3D"">ServletInitialHandler$1.<wbr = class=3D"">handleRequest(<wbr class=3D"">ServletInitialHandler.java:<wbr = class=3D"">104) at io.undertow.server.Connectors.<wbr = class=3D"">executeRootHandler(Connectors.<wbr class=3D"">java:202) = at io.undertow.server.<wbr class=3D"">HttpServerExchange$1.run(<wbr = class=3D"">HttpServerExchange.java:805) at = java.util.concurrent.<wbr class=3D"">ThreadPoolExecutor.runWorker(<wbr = class=3D"">ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at = java.util.concurrent.<wbr class=3D"">ThreadPoolExecutor$Worker.run(<wbr = class=3D"">ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at = java.lang.Thread.run(Thread.<wbr class=3D"">java:745) [rt.jar:1.8.0_111] = 2016-12-20 16:53:57,859-05 ERROR [org.ovirt.engine.core.<wbr = class=3D"">vdsbroker.vdsbroker.<wbr class=3D"">PollVDSCommand] (default = task-17) [5ecd5a55-2b7a-4dd6-b42b-<wbr class=3D"">cc49bbfb3962] Timeout = waiting for VDSM response: Internal timeout occured <br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>> This is definitely not related to the patch = linked above.<br class=3D""> >>>>>>><br class=3D""> >>>>>>> However I am also not quite sure what is = the cause of this right now, it might be a breaking change in VDSM.<br = class=3D""> >>>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>> The DC level is 4.0, is it possible that is the = reason for the failure? <br class=3D""> >>>>>><br class=3D""> >>>>>> <br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> Full test code can be seen here:<br = class=3D""> >>>>>>>> <a = href=3D"https://gerrit.ovirt.org/gitweb?p=3Dovirt-system-tests.git;a=3Dblo= b;f=3Dbasic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=3Drefs/heads/m= aster#l291" target=3D"_blank" class=3D"">https://gerrit.ovirt.org/<wbr = class=3D"">gitweb?p=3Dovirt-system-tests.<wbr = class=3D"">git;a=3Dblob;f=3Dbasic-suite-4.1/<wbr = class=3D"">test-scenarios/004_basic_<wbr = class=3D"">sanity.py;hb=3Drefs/heads/<wbr class=3D"">master#l291</a><br = class=3D""> >>>>>>>><br class=3D""> >>>>>>>> Full test exception can be seen = here:<br class=3D""> >>>>>>>> <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/t= estReport/junit/%28root%29/004_basic_sanity/vm_run/" target=3D"_blank" = class=3D"">http://jenkins.ovirt.org/job/<wbr = class=3D"">test-repo_ovirt_experimental_<wbr = class=3D"">4.1/13/testReport/junit/%<wbr = class=3D"">28root%29/004_basic_sanity/vm_<wbr class=3D"">run/</a><br = class=3D""> >>>>>>>><br class=3D""> >>>>>>>> Further logs can be seen in Jenkins:<br = class=3D""> >>>>>>>> <a = href=3D"http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/a= rtifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_= logs/basic-suite-4.1/post-004_basic_sanity.py/" target=3D"_blank" = class=3D"">http://jenkins.ovirt.org/job/<wbr = class=3D"">test-repo_ovirt_experimental_<wbr = class=3D"">4.1/13/artifact/exported-<wbr = class=3D"">artifacts/basic_suite_4.1.sh-<wbr = class=3D"">el7/exported-artifacts/test_<wbr = class=3D"">logs/basic-suite-4.1/post-004_<wbr = class=3D"">basic_sanity.py/</a><br class=3D""> >>>>>>>><br class=3D""> >>>>>>>> -- <br class=3D""> >>>>>>>> Barak Korren<br class=3D""> >>>>>>>> <a href=3D"mailto:bkorren@redhat.com" = target=3D"_blank" class=3D"">bkorren@redhat.com</a><br class=3D""> >>>>>>>> RHCE, RHCi, RHV-DevOps Team<br = class=3D""> >>>>>>>> <a = href=3D"https://ifireball.wordpress.com/" target=3D"_blank" = class=3D"">https://ifireball.wordpress.<wbr class=3D"">com/</a><br = class=3D""> >>>>>>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>>>>>> Devel mailing list<br class=3D""> >>>>>>>> <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D""> >>>>>>>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>><br class=3D""> >>>>>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>>>>> Devel mailing list<br class=3D""> >>>>>>> <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D""> >>>>>>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>><br class=3D""> >>>>>> -- <br class=3D""> >>>>>> Eyal Edri<br class=3D""> >>>>>> Associate Manager<br class=3D""> >>>>>> RHV DevOps<br class=3D""> >>>>>> EMEA ENG Virtualization R&D<br class=3D""> >>>>>> Red Hat Israel<br class=3D""> >>>>>><br class=3D""> >>>>>> phone: <a href=3D"tel:+972%209-769-2018" = value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br = class=3D""> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br = class=3D""> >>>>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>>>> Devel mailing list<br class=3D""> >>>>>> <a href=3D"mailto:Devel@ovirt.org" = target=3D"_blank" class=3D"">Devel@ovirt.org</a><br class=3D""> >>>>>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>>>><br class=3D""> >>>>><br class=3D""> >>>><br class=3D""> >>>><br class=3D""> >>>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >>>> Devel mailing list<br class=3D""> >>>> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> >>>> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel"= target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""> >>><br class=3D""> >>><br class=3D""> >><br class=3D""> ><br class=3D""> ><br class=3D""> ><br class=3D""> > -- <br class=3D""> > Eyal Edri<br class=3D""> > Associate Manager<br class=3D""> > RHV DevOps<br class=3D""> > EMEA ENG Virtualization R&D<br class=3D""> > Red Hat Israel<br class=3D""> ><br class=3D""> > phone: <a href=3D"tel:+972%209-769-2018" value=3D"+97297692018" = target=3D"_blank" class=3D"">+972-9-7692018</a><br class=3D""> > irc: eedri (on #tlv #rhev-dev #rhev-integ)<br class=3D""> ><br class=3D""> > ______________________________<wbr class=3D"">_________________<br = class=3D""> > Devel mailing list<br class=3D""> > <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></p> </blockquote></div></div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_4A67E93F-035B-42BA-9727-09586A1C7969--