job test-repo_ovirt_experimental_4.0 (3718) failed
Hi, Job [1] failed with the following errors (logs [2]): {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}}� 2017-01-19 12:13:51,619 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [3a116b94] Message received: {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}} 2017-01-19 12:13:51,620 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Failed in 'AttachStorageDomainVDS' method 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [6a6480a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',) 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='fc33da6d-5da7-4005-a693-f170437c176c', ignoreFailoverLimit='false', storageDomainId='ab6c9588-d957-4be3-9862-d2596db463d9'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 2017-01-19 12:13:51,632 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler10) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.hostsStorageConnectionsAndPoolMetadataRefresh#-9223372036854775793 as there is no unfired trigger. 2017-01-19 12:13:51,624 DEBUG [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.createDefaultConcreteException(AttachStorageDomainVDSCommand.java:25) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:192) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.executeIrsBrokerCommand(AttachStorageDomainVDSCommand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.lambda$executeVDSCommand$0(IrsBrokerCommand.java:161) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.runInControlledConcurrency(IrsProxyData.java:248) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:158) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand.executeCommand(AttachStorageDomainToPoolCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:428) [bll.jar:] at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:229) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:143) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:133) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:152) at org.ovirt.engine.api.restapi.resource.BackendAttachedStorageDomainsResource.add(BackendAttachedStorageDomainsResource.java:63) at org.ovirt.engine.api.v3.V3Server.adaptAdd(V3Server.java:166) at org.ovirt.engine.api.v3.servers.V3AttachedStorageDomainsServer.add(V3AttachedStorageDomainsServer.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:84) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:63) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/ [2] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/artifact/... Best Regards, Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1
Hi, I see some issues in the communication between host0 and the the engine. Vdsm returned the answer to call getSpmStatus and then got an SSL error. [1] After that, the engine tried to send to the same host0 connectStorageServer requests that failed. The process of attaching the storage domain failed because the host was not connected to the NFS server. I think the engine should not have tried to attach the storage domain if connectStorageServer request failed. Can you open a bug on this ? Regardless, the SSL error and communication issue should be investigated further by infra team. Regards, Fred [1] jsonrpc.Executor/5::DEBUG::2017-01-19 12:13:48,711::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2L} jsonrpc.Executor/5::INFO::2017-01-19 12:13:48,711::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-01-19 12:13:49,449::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof [2] 2017-01-19 12:13:49,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [4a2c0e44] Command 'ConnectStorageServerVDSCommand(HostName = lago-basic-suite-4-0-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='4c4d9ad9-32a1-406a-93da-1710e069c690', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da9131c8-1cd9-46bb-b7ea-f83228e04d3f', connection='192.168.201.4:/exports/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues On Mon, Jan 23, 2017 at 4:35 PM, Shlomo Ben David <sbendavi@redhat.com> wrote:
Hi,
Job [1] failed with the following errors (logs [2]):
{"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}}� 2017-01-19 12:13:51,619 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [3a116b94] Message received: {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}} 2017-01-19 12:13:51,620 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Failed in 'AttachStorageDomainVDS' method 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [6a6480a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',) 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='fc33da6d-5da7-4005-a693-f170437c176c', ignoreFailoverLimit='false', storageDomainId='ab6c9588-d957-4be3-9862-d2596db463d9'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 2017-01-19 12:13:51,632 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler10) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.hostsStorageConnectionsAndPoolMetadataRefresh#-9223372036854775793 as there is no unfired trigger. 2017-01-19 12:13:51,624 DEBUG [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.createDefaultConcreteException(AttachStorageDomainVDSCommand.java:25) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:192) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.executeIrsBrokerCommand(AttachStorageDomainVDSCommand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.lambda$executeVDSCommand$0(IrsBrokerCommand.java:161) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.runInControlledConcurrency(IrsProxyData.java:248) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:158) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand.executeCommand(AttachStorageDomainToPoolCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:428) [bll.jar:] at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:229) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:143) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:133) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:152) at org.ovirt.engine.api.restapi.resource.BackendAttachedStorageDomainsResource.add(BackendAttachedStorageDomainsResource.java:63) at org.ovirt.engine.api.v3.V3Server.adaptAdd(V3Server.java:166) at org.ovirt.engine.api.v3.servers.V3AttachedStorageDomainsServer.add(V3AttachedStorageDomainsServer.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:84) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:63) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
[1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/ [2] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/ artifact/exported-artifacts/basic_suite_4.0.sh-el7/ exported-artifacts/test_logs/basic-suite-4.0/post-002_bootstrap.py/
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Tue, Jan 24, 2017 at 9:38 AM, Fred Rolland <frolland@redhat.com> wrote:
Hi,
I see some issues in the communication between host0 and the the engine. Vdsm returned the answer to call getSpmStatus and then got an SSL error. [1]
After that, the engine tried to send to the same host0 connectStorageServer requests that failed. The process of attaching the storage domain failed because the host was not connected to the NFS server. I think the engine should not have tried to attach the storage domain if connectStorageServer request failed. Can you open a bug on this ?
Regardless, the SSL error and communication issue should be investigated further by infra team.
Regards,
Fred
[1] jsonrpc.Executor/5::DEBUG::2017-01-19 12:13:48,711::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2L} jsonrpc.Executor/5::INFO::2017-01-19 12:13:48,711::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-01-19 12:13:49,449::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
The above log entry is a result of the engine closing connection due to higher level timeout occurring
[2] 2017-01-19 12:13:49,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [4a2c0e44] Command 'ConnectStorageServerVDSCommand(HostName = lago-basic-suite-4-0-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='4c4d9ad9-32a1-406a-93da-1710e069c690', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da9131c8-1cd9-46bb-b7ea-f83228e04d3f', connection='192.168.201.4:/exports/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
Above log entry tells us that a command timeout. In order to understand what happened I would check this command why it did not get a response. Usually it is send 300 secs prior above message. It is desired approach when we see higher timeout to close the connection and attempt to reconnect.
On Mon, Jan 23, 2017 at 4:35 PM, Shlomo Ben David <sbendavi@redhat.com> wrote:
Hi,
Job [1] failed with the following errors (logs [2]):
{"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}}� 2017-01-19 12:13:51,619 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [3a116b94] Message received: {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}} 2017-01-19 12:13:51,620 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Failed in 'AttachStorageDomainVDS' method 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [6a6480a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',) 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='fc33da6d-5da7-4005-a693-f170437c176c', ignoreFailoverLimit='false', storageDomainId='ab6c9588-d957-4be3-9862-d2596db463d9'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 2017-01-19 12:13:51,632 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler10) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.hostsStorageConnectionsAndPoolMetadataRefresh#-9223372036854775793 as there is no unfired trigger. 2017-01-19 12:13:51,624 DEBUG [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.createDefaultConcreteException(AttachStorageDomainVDSCommand.java:25) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:192) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand.executeIrsBrokerCommand(AttachStorageDomainVDSCommand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.lambda$executeVDSCommand$0(IrsBrokerCommand.java:161) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.runInControlledConcurrency(IrsProxyData.java:248) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:158) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:657) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand.executeCommand(AttachStorageDomainToPoolCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:428) [bll.jar:] at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:229) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:143) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:133) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:152) at org.ovirt.engine.api.restapi.resource.BackendAttachedStorageDomainsResource.add(BackendAttachedStorageDomainsResource.java:63) at org.ovirt.engine.api.v3.V3Server.adaptAdd(V3Server.java:166) at org.ovirt.engine.api.v3.servers.V3AttachedStorageDomainsServer.add(V3AttachedStorageDomainsServer.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:84) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:63) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
[1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/ [2] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3718/artifact/...
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Tue, Jan 24, 2017 at 11:09 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Tue, Jan 24, 2017 at 9:38 AM, Fred Rolland <frolland@redhat.com> wrote:
Hi,
I see some issues in the communication between host0 and the the engine. Vdsm returned the answer to call getSpmStatus and then got an SSL error. [1]
After that, the engine tried to send to the same host0 connectStorageServer requests that failed. The process of attaching the storage domain failed because the host was not connected to the NFS server. I think the engine should not have tried to attach the storage domain if connectStorageServer request failed. Can you open a bug on this ?
Regardless, the SSL error and communication issue should be investigated further by infra team.
Regards,
Fred
[1] jsonrpc.Executor/5::DEBUG::2017-01-19 12:13:48,711::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2L} jsonrpc.Executor/5::INFO::2017-01-19 12:13:48,711::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-01-19 12:13:49,449::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
The above log entry is a result of the engine closing connection due to higher level timeout occurring
[2] 2017-01-19 12:13:49,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStorageServerVDSCommand]
(org.ovirt.thread.pool-6-thread-5) [4a2c0e44] Command 'ConnectStorageServerVDSCommand(HostName = lago-basic-suite-4-0-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='4c4d9ad9-32a1-406a-93da-1710e069c690', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da9131c8-1cd9-46bb-b7ea- f83228e04d3f', connection='192.168.201.4:/exports/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
Above log entry tells us that a command timeout. In order to understand what happened I would check this command why it did not get a response. Usually it is send 300 secs prior above message.
It is desired approach when we see higher timeout to close the connection and attempt to reconnect.
Is there a reason not to close gracefully? Y.
On Mon, Jan 23, 2017 at 4:35 PM, Shlomo Ben David <sbendavi@redhat.com> wrote:
Hi,
Job [1] failed with the following errors (logs [2]):
{"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8",
{"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}}� 2017-01-19 12:13:51,619 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [3a116b94] Message received: {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}} 2017-01-19 12:13:51,620 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker. AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Failed in 'AttachStorageDomainVDS' method 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [6a6480a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',) 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker. AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='fc33da6d-5da7-4005-a693-f170437c176c', ignoreFailoverLimit='false', storageDomainId='ab6c9588-d957-4be3-9862-d2596db463d9'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 2017-01-19 12:13:51,632 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler10) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData. hostsStorageConnectionsAndPoolMetadataRefresh#-9223372036854775793 as there is no unfired trigger. 2017-01-19 12:13:51,624 DEBUG [org.ovirt.engine.core.vdsbroker.irsbroker. AttachStorageDomainVDSCommand] (default task-27) [6a6480a4] Exception: org.ovirt.engine.core.vdsbroker.irsbroker. IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 at org.ovirt.engine.core.vdsbroker.irsbroker. AttachStorageDomainVDSCommand.createDefaultConcreteException( AttachStorageDomainVDSCommand.java:25) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase. createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.
[vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker. AttachStorageDomainVDSCommand.executeIrsBrokerCommand( AttachStorageDomainVDSCommand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.lambda$ executeVDSCommand$0(IrsBrokerCommand.java:161) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData. runInControlledConcurrency(IrsProxyData.java:248) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand. executeVDSCommand(IrsBrokerCommand.java:158) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase. executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute( VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager. runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand( VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand( CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase. runVdsCommand(StorageHandlingCommandBase.java:657) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolComma nd.executeCommand(AttachStorageDomainToPoolCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction( CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScop e(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction( CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction( CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:428) [bll.jar:] at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodIntercep tor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception( Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception( Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation( Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.
at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.
at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerIntercepto r.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.
at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.
at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat ionInterceptor.aroundInvoke(AbstractEJBRequestScopeActivat ionInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter ceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.
[wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation( InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor .processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAsso ciationInterceptor.processInvocation(SingletonComponentInstanceAsso ciationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx( CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports( CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation( CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterc eptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.
"error": proceedProxyReturnValue(BrokerCommandBase.java:192) proceed(InterceptorContext.java:437) processInvocation(UserInterceptorFactory.java:63) proceed(InterceptorContext.java:437) processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) processInvocation(ExecutionTimeInterceptor.java:43) proceed(InterceptorContext.java:437) processInvocation(ConcurrentContextInterceptor.java:45) processInvocation(WaitTimeInterceptor.java:47) processInvocation(SecurityContextInterceptor.java:100) processInvocation(StartupAwaitInterceptor.java:22) processInvocation(ShutDownInterceptorFactory.java:64) processInvocation(LoggingInterceptor.java:67) processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation( ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation( AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerIntercep tor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation( ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke( ProxyInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$ view4.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource. doAction(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResou rce.doCreateEntity(AbstractBackendCollectionResource.java:229) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResou rce.performCreate(AbstractBackendCollectionResource.java:143) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResou rce.performCreate(AbstractBackendCollectionResource.java:133) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResou rce.performCreate(AbstractBackendCollectionResource.java:152) at org.ovirt.engine.api.restapi.resource.BackendAttachedStorageDomainsR esource.add(BackendAttachedStorageDomainsResource.java:63) at org.ovirt.engine.api.v3.V3Server.adaptAdd(V3Server.java:166) at org.ovirt.engine.api.v3.servers.V3AttachedStorageDomainsServer.add( V3AttachedStorageDomainsServer.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke( MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject( ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke( ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject( ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke( ResourceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject( ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke( ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke( SynchronousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke( SynchronousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher. service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet. HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet. HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest( ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest( FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler. handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest( ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath( ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl( RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup( RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward( RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:84) at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:63) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter( RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter. doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter( SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter. doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter( SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter .doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter. doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter. doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter. doFilter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest( FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler. handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest( ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHand ler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandl er.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandl er.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandle r.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler .handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstrai ntHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandl er.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandle r.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHand ler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler. handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssocia tionHandler.handleRequest(AbstractSecurityContextAssocia tionHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler. handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest( ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call( ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call( ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest( ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors. java:202) at io.undertow.server.HttpServerExchange$1.run( HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
[1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.0/3718/ [2] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.0/3718/artifact/exported-artifacts/basic_suite_4.0.sh- el7/exported-artifacts/test_logs/basic-suite-4.0/post-002_bootstrap.py/
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Please let me know if I should or shouldn't open a bug and on which product. Thanks Gil On Tue, Jan 24, 2017 at 12:00 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Tue, Jan 24, 2017 at 11:09 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Tue, Jan 24, 2017 at 9:38 AM, Fred Rolland <frolland@redhat.com> wrote:
Hi,
I see some issues in the communication between host0 and the the engine. Vdsm returned the answer to call getSpmStatus and then got an SSL error. [1]
After that, the engine tried to send to the same host0 connectStorageServer requests that failed. The process of attaching the storage domain failed because the host was not connected to the NFS server. I think the engine should not have tried to attach the storage domain if connectStorageServer request failed. Can you open a bug on this ?
Regardless, the SSL error and communication issue should be investigated further by infra team.
Regards,
Fred
[1] jsonrpc.Executor/5::DEBUG::2017-01-19 12:13:48,711::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 2L} jsonrpc.Executor/5::INFO::2017-01-19 12:13:48,711::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-01-19 12:13:49,449::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
The above log entry is a result of the engine closing connection due to higher level timeout occurring
[2] 2017-01-19 12:13:49,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageSer
verVDSCommand]
(org.ovirt.thread.pool-6-thread-5) [4a2c0e44] Command 'ConnectStorageServerVDSCommand(HostName = lago-basic-suite-4-0-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='4c4d9ad9-32a1-406a-93da-1710e069c690', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da9131c8- 1cd9-46bb-b7ea-f83228e04d3f', connection='192.168.201.4:/exports/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
Above log entry tells us that a command timeout. In order to understand what happened I would check this command why it did not get a response. Usually it is send 300 secs prior above message.
It is desired approach when we see higher timeout to close the connection and attempt to reconnect.
Is there a reason not to close gracefully? Y.
On Mon, Jan 23, 2017 at 4:35 PM, Shlomo Ben David <sbendavi@redhat.com> wrote:
Hi,
Job [1] failed with the following errors (logs [2]):
{"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8",
{"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}}� 2017-01-19 12:13:51,619 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [3a116b94] Message received: {"jsonrpc": "2.0", "id": "669e1306-3206-4a64-a33f-d18176531ff8", "error": {"message": "Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',)", "code": 358}} 2017-01-19 12:13:51,620 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDoma inVDSCommand] (default task-27) [6a6480a4] Failed in 'AttachStorageDomainVDS' method 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [6a6480a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',) 2017-01-19 12:13:51,624 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDoma inVDSCommand] (default task-27) [6a6480a4] Command 'AttachStorageDomainVDSCommand( AttachStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='fc33da6d-5da7-4005-a693-f170437c176c', ignoreFailoverLimit='false', storageDomainId='ab6c9588-d957-4be3-9862-d2596db463d9'})' execution failed: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 2017-01-19 12:13:51,632 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler10) [] Rescheduling DEFAULT.org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyDa ta.hostsStorageConnectionsAndPoolMetadataRefresh#-9223372036854775793 as there is no unfired trigger. 2017-01-19 12:13:51,624 DEBUG [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDoma inVDSCommand] (default task-27) [6a6480a4] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailed NoFailoverException: IRSGenericException: IRSErrorException: Failed to AttachStorageDomainVDS, error = Storage domain does not exist: (u'ab6c9588-d957-4be3-9862-d2596db463d9',), code = 358 at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomai nVDSCommand.createDefaultConcreteException(AttachStorageDoma inVDSCommand.java:25) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase. createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.
[vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomai nVDSCommand.executeIrsBrokerCommand(AttachStorageDomainVDSCo mmand.java:18) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand. lambda$executeVDSCommand$0(IrsBrokerCommand.java:161) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData.runIn ControlledConcurrency(IrsProxyData.java:248) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.e xecuteVDSCommand(IrsBrokerCommand.java:158) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeComman d(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandB ase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsComman d(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsComman d(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandB ase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase .runVdsCommand(StorageHandlingCommandBase.java:657) [bll.jar:] at org.ovirt.engine.core.bll.storage.domain.AttachStorageDomain ToPoolCommand.executeCommand(AttachStorageDomainToPoolCommand.java:152) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransact ion(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransac tionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(Comma ndBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.e xecuteInSuppressed(TransactionSupport.java:166) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.e xecuteInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase. java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandB ase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:475) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:428) [bll.jar:] at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
[rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.
at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInte rception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInte rception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvoc ation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactor y$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerI nterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
[rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInt erceptor.processInvocation(ManagedReferenceLifecycleMethodIn terceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeI nterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationIntercep tor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.
[wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.proc essInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(In itialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherIn terceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInst anceAssociationInterceptor.processInvocation(SingletonCompon entInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInte rceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxIntercep tor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTT xInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationCo ntextInterceptor.processInvocation(CurrentInvoc ationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterc eptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.proces sInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitIntercep tor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptor Factory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.
[wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.proces sInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processIn vocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorConte xt.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecke d(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvoca tion(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorConte xt.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.
at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewServic e.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocatio n(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(Prox yInvocationHandler.java:73) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4 .runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doActi on(BackendResource.java:250) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollect ionResource.doCreateEntity(AbstractBackendCollectionResource.java:229) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollect ionResource.performCreate(AbstractBackendCollectionResource.java:143) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollect ionResource.performCreate(AbstractBackendCollectionResource.java:133) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollect ionResource.performCreate(AbstractBackendCollectionResource.java:152) at org.ovirt.engine.api.restapi.resource.BackendAttachedStorage DomainsResource.add(BackendAttachedStorageDomainsResource.java:63) at org.ovirt.engine.api.v3.V3Server.adaptAdd(V3Server.java:166) at org.ovirt.engine.api.v3.servers.V3AttachedStorageDomainsServ er.add(V3AttachedStorageDomainsServer.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:62) [rt.jar:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
[rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInje ctorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget (ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(Resourc eMethodInvoker.java:249) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge tObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(Resour ceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge tObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(Resour ceLocatorInvoker.java:107) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge tObject(ResourceLocatorInvoker.java:133) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(Resour ceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchro nousDispatcher.java:402) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(Synchro nousDispatcher.java:209) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDi spatcher.service(ServletContainerDispatcher.java:221) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc her.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc her.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(Se rvletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(Fil terHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHan
at io.undertow.servlet.handlers.ServletDispatchingHandler.handl eRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchR equest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchT oPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(R equestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSe tup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(Reque stDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilt er(VersionFilter.java:139) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilt er(VersionFilter.java:68) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilt er(CurrentFilter.java:84) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilt er(CurrentFilter.java:63) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.d oFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter (EnforceAuthFilter.java:39) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilte r.doFilter(SsoRestApiNegotiationFilter.java:91) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFil ter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.do Filter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFi lter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.d oFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.d oFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFi lter(CORSSupportFilter.java:183) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilte r.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d oFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(Fil terHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHan
"error": proceedProxyReturnValue(BrokerCommandBase.java:192) thodAccessorImpl.java:43) processInvocation(ManagedReferenceMethodInterceptor.java:52) thodAccessorImpl.java:43) processInvocation(EjbRequestScopeActivationInterceptor.java:83) processInvocation(LoggingInterceptor.java:67) processInvocation(PrivilegedWithCombinerInterceptor.java:80) thodAccessorImpl.java:43) dler.handleRequest(ServletSecurityRoleHandler.java:62) dler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handl eRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssoc iationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociat ionHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationC allHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandle r.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler .handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentiality ConstraintHandler.handleRequest(ServletConfident ialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstra intHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandle r.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSes sionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.ha ndleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssocia tionHandler.handleRequest(AbstractSecurityContextAssociation Handler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHa ndler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(P redicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFir stRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$10 0(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(Se rvletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(Se rvletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAct ion$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1. call(ContextClassLoaderSetupAction.java:43) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1. call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1. call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1. call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1. call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchR equest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$00 0(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleR equest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors. java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchan ge.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool Executor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
[1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. 0/3718/ [2] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. 0/3718/artifact/exported-artifacts/basic_suite_4.0.sh-el7/ exported-artifacts/test_logs/basic-suite-4.0/post-002_bootstrap.py/
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (5)
-
Fred Rolland -
Gil Shinar -
Piotr Kliczewski -
Shlomo Ben David -
Yaniv Kaul