[ovirt-devel] [ oVirt Devel ] [oVirt master ] [ OST Failure Report ] [ basic_suite_master & upgrade_suites failed ]

*Test Failed:* 1. *basic_suite: *002_bootstrap.add_dc and 2. *upgrade_from_release: *001_upgrade_engine.test_initialize_engine 3. *upgrade_from_prevrelease*: *Link to test logs:* 1. *basic suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifa... 2. *upgrade from prev release suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifa... 3. *upgrade from release suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifa... *Suspected patch: https://gerrit.ovirt.org/#/c/79033/41 <https://gerrit.ovirt.org/#/c/79033/41>* *Please note that this change was the only change under those tests.* *Error snippet from basic-suite /var/log/ovirt-engine/server.log:* ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /ovirt-engine/api/v4/datacenters: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:245) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:180) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:284) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141] Caused by: org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:78) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:222) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:179) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:422) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] ... 73 more Caused by: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at org.ovirt.engine.api.resource.DataCentersResource.doAdd(DataCentersResource.java:58) [restapi-definition.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] ... 88 more Caused by: java.lang.ClassNotFoundException: org.ovirt.api.metamodel.server.ValidationException from [Module "org.ovirt.engine.api.restapi-definition" from local module loader @61064425 (finder: local module finder @7b1d7fff (roots: /usr/share/ovirt-engine-wildfly-overlay/modules,/usr/share/ovirt-engine/modules/common,/usr/share/ovirt-engine-extension-aaa-jdbc/modules,/usr/share/ovirt-engine-extension-aaa-ldap/modules,/usr/share/ovirt-engine-wildfly/modules,/usr/share/ovirt-engine-wildfly/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:412) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:400) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116) [jboss-modules.jar:1.6.0.CR2] ... 99 more *Error snippet from both upgrade suites (engine setup output):*[ ERROR ] Failed to execute stage 'Closing up': Command '/bin/firewall-cmd' failed to execute -- DANIEL BELENKY RHV DEVOPS Red Hat EMEA <https://www.redhat.com/> IRC: #rhev-integ #rhev-dev <https://red.ht/sig>