There are two issues here:
1. OST is exiting with wring error due to the local function not working which Gal has a patch for
2. there is an actual code regression which we suspects comes from the SDK

We suspect the issue is a new sdk package build yesterday
https://cbs.centos.org/koji/buildinfo?buildID=23581

Here is a link to the fist failure's logs:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8800/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/

Dominic is currently looking at the issue.

<error>

2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] Got: ***L:INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64
2018-07-26 13:47:37,745-04 DEBUG [org.ovirt.otopi.dialog.MachineDialogParser] (VdsDeploy) [3283d2df] nextEvent: Log INFO Yum install: 217/529: libosinfo-1.0.0-1.el7.x86_64
2018-07-26 13:47:37,754-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [b4915769-1cf0-4526-9214-e932d078cf07] method: runAction, params: [TestProviderConnectivity, ProviderParameters:{comma
ndId='d29721ff-3dd7-4932-b43d-eee819f1afee', user='null', commandType='Unknown'}], timeElapsed: 33ms
2018-07-26 13:47:37,766-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [3283d2df] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-basic-suite-master-host-1. Yum install: 217/529: l
ibosinfo-1.0.0-1.el7.x86_64.
2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: WFLYEJB0442: Unexpected Error
2018-07-26 13:47:37,782-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Exception: javax.ejb.EJBException: WFLYEJB0442: Unexpected Error
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:218) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
        at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
        at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-13.0.0.Final.jar:13.0.0.Final]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
        at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:619)
        at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
        at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runAction(Unknown Source)
        at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) [restapi-jaxrs.jar:]
        at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) [restapi-jaxrs.jar:]
        at org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:174) [restapi-jaxrs.jar:]
        at org.ovirt.engine.api.restapi.resource.AbstractBackendExternalProviderResource.testConnectivity(AbstractBackendExternalProviderResource.java:46) [restapi-jaxrs.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]

<error>

On Fri, Jul 27, 2018 at 11:58 AM, Dafna Ron <dron@redhat.com> wrote:
it seems that the change to the test was done a while back and we did not hit it since Glance was available.



On Fri, Jul 27, 2018 at 11:52 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,

We seem to have a problem in the OST tests that is causing failure in multiple projects

The issue is that setting __name__ on local function is not working correctly and we are exiting instead of skipping when Glance image is not available.

We had the same issue on a different test where Milan fixed it: https://gerrit.ovirt.org/#/c/93167/
but now it seems that we have the same issue on list_glace_images as well.
 

ERROR: 

Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test
    test()
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper
    return func(get_test_prefix(), *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 78, in wrapper
    prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs
  File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 633, in list_glance_images
    raise SkipTest('%s: GLANCE connectivity test failed' % list_glance_images_4.__name__ )
NameError: global name 'list_glance_images_4' is not defined