[ OST Failure Report ] [ oVirt Master (ALL) ] [ 27-07-2018 ] [ 002_bootstrap.list_glance_images ]
Hi, We seem to have a problem in the OST tests that is causing failure in multiple projects The issue is that setting __name__ on local function is not working correctly and we are exiting instead of skipping when Glance image is not available. We had the same issue on a different test where Milan fixed it: https://gerrit.ovirt.org/#/c/93167/ but now it seems that we have the same issue on list_glace_images as well. ERROR: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 633, in list_glance_images raise SkipTest('%s: GLANCE connectivity test failed' % list_glance_images_4.__name__ ) NameError: global name 'list_glance_images_4' is not defined
Gal posted a fix: https://gerrit.ovirt.org/#/c/93350/ it seems that the change to the test was done a while back and we did not hit it since Glance was available. On Fri, Jul 27, 2018 at 11:52 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We seem to have a problem in the OST tests that is causing failure in multiple projects
The issue is that setting __name__ on local function is not working correctly and we are exiting instead of skipping when Glance image is not available.
We had the same issue on a different test where Milan fixed it: https://gerrit.ovirt.org/#/c/93167/ but now it seems that we have the same issue on list_glace_images as well.
ERROR:
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 633, in list_glance_images raise SkipTest('%s: GLANCE connectivity test failed' % list_glance_images_4.__name__ ) NameError: global name 'list_glance_images_4' is not defined
There are two issues here: 1. OST is exiting with wring error due to the local function not working which Gal has a patch for 2. there is an actual code regression which we suspects comes from the SDK We suspect the issue is a new sdk package build yesterday https://cbs.centos.org/koji/buildinfo?buildID=23581 Here is a link to the fist failure's logs: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8800/artifact... Dominic is currently looking at the issue. <error> 2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] Got: ***L:INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64 2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] nextEvent: Log INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64 2018-07-26 13:47:37,754-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [b4915769-1cf0-4526-9214-e932d078cf07] method: runAction, params: [TestProviderConnectivity, ProviderParameters:{comma ndId='d29721ff-3dd7-4932-b43d-eee819f1afee', user='null', commandType='Unknown'}], timeElapsed: 33ms 2018-07-26 13:47:37,766-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [3283d2df] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-1. Yum install: 217/529: l ibosinfo-1.0.0-1.el7.x86_64. 2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: WFLYEJB0442: Unexpected Error 2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Exception: javax.ejb.EJBException: WFLYEJB0442: Unexpected Error at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:218) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runAction(Unknown Source) at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:174) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendExternalProviderResource.testConnectivity(AbstractBackendExternalProviderResource.java:46) [restapi-jaxrs.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181] <error> On Fri, Jul 27, 2018 at 11:58 AM, Dafna Ron <dron@redhat.com> wrote:
Gal posted a fix: https://gerrit.ovirt.org/#/c/93350/
it seems that the change to the test was done a while back and we did not hit it since Glance was available.
On Fri, Jul 27, 2018 at 11:52 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We seem to have a problem in the OST tests that is causing failure in multiple projects
The issue is that setting __name__ on local function is not working correctly and we are exiting instead of skipping when Glance image is not available.
We had the same issue on a different test where Milan fixed it: https://gerrit.ovirt.org/#/c/93167/ but now it seems that we have the same issue on list_glace_images as well.
ERROR:
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 633, in list_glance_images raise SkipTest('%s: GLANCE connectivity test failed' % list_glance_images_4.__name__ ) NameError: global name 'list_glance_images_4' is not defined
On Fri, 27 Jul 2018 15:37:23 +0100 Dafna Ron <dron@redhat.com> wrote:
There are two issues here: 1. OST is exiting with wring error due to the local function not working which Gal has a patch for 2. there is an actual code regression which we suspects comes from the SDK
We suspect the issue is a new sdk package build yesterday https://cbs.centos.org/koji/buildinfo?buildID=23581
Here is a link to the fist failure's logs: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8800/artifact...
Dominic is currently looking at the issue.
One reason was an incompatibility on bytecode level of the openstack-java-sdk used at compile time and runtime. Fix posted on https://gerrit.ovirt.org/93352
<error>
2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] Got: ***L:INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64 2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] nextEvent: Log INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64 2018-07-26 13:47:37,754-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [b4915769-1cf0-4526-9214-e932d078cf07] method: runAction, params: [TestProviderConnectivity, ProviderParameters:{comma ndId='d29721ff-3dd7-4932-b43d-eee819f1afee', user='null', commandType='Unknown'}], timeElapsed: 33ms 2018-07-26 13:47:37,766-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [3283d2df] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-1. Yum install: 217/529: l ibosinfo-1.0.0-1.el7.x86_64. 2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: WFLYEJB0442: Unexpected Error 2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Exception: javax.ejb.EJBException: WFLYEJB0442: Unexpected Error at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:218) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runAction(Unknown Source) at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:174) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendExternalProviderResource.testConnectivity(AbstractBackendExternalProviderResource.java:46) [restapi-jaxrs.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
<error>
On Fri, Jul 27, 2018 at 11:58 AM, Dafna Ron <dron@redhat.com> wrote:
Gal posted a fix: https://gerrit.ovirt.org/#/c/93350/
it seems that the change to the test was done a while back and we did not hit it since Glance was available.
On Fri, Jul 27, 2018 at 11:52 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We seem to have a problem in the OST tests that is causing failure in multiple projects
The issue is that setting __name__ on local function is not working correctly and we are exiting instead of skipping when Glance image is not available.
We had the same issue on a different test where Milan fixed it: https://gerrit.ovirt.org/#/c/93167/ but now it seems that we have the same issue on list_glace_images as well.
ERROR:
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 633, in list_glance_images raise SkipTest('%s: GLANCE connectivity test failed' % list_glance_images_4.__name__ ) NameError: global name 'list_glance_images_4' is not defined
participants (2)
-
Dafna Ron -
Dominik Holler