On Thu, Feb 28, 2019 at 12:01 PM Dafna Ron <dron@redhat.com> wrote:
Hi,

The test failed due to not finding master storage domain
I am seeing a build of engine that passed so its possible that this is related to the random failures we have been having on other tests but since the change reported is storage related, can you please review the logs to see if anything happened to the master domain while trying to resize?

First the patch cannot affect the storage domain, this is just moving of
constant from one module to another, and verified by vdsm tests. If the patch
was broken we would fail with a NameError in vdsm, accessing non-exisitng
variable.

I think it is a flaky test, since I ran many successful OST with this patch.
These run include the patch:
- https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4173/
- https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4172/
- https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4171/

There may be real issue behind this, but it is not related to the patch.

I think this is related to not deactivating lvs when putting storage domain in maintenance.
We have this bug for the issue:
https://bugzilla.redhat.com/1163890

The issue is probably random since storage domain monitor is refreshed every 5 minutes,
so the system correct itself automatically. It probably fails if the test happened to start at the 
wrong time, timing out before the system refreshed the storage.

If you see more of this issue, please file a new bug and we will handle it when we have time.
In the meantime you can disable the test since it is not reliable.

Nir


CQ reported this change as root cause:
https://gerrit.ovirt.org/#/c/98015/6 - blockSD: Move legacy block size constants to sd

build failure:

logs:

errors in log:

2019-02-27 17:35:02,222-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM la
go-basic-suite-master-host-1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9'
2019-02-27 17:35:02,222-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5
a9']]'
2019-02-27 17:35:02,222-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] HostName = lago-basic-suite-master-host-1
2019-02-27 17:35:02,223-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command 'ConnectStoragePoolVDSCommand(HostName = lago-basic-suite-master-host-1, ConnectStoragePoolVDSCommandParameters:{hostId='1ac0eda2-0980-40df-b029-b262680696a4', vdsId='1ac0eda2-0980-40df-b029-b262680696a4', storagePoolId='1c79b83e-32d6-43e0-bc61-2955e42c3a58', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9'
2019-02-27 17:35:02,223-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Exception: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9'
        at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:50) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedConnectProxyReturnValue(ConnectStoragePoolVDSCommand.java:48) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedProxyReturnValue(ConnectStoragePoolVDSCommand.java:36) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.executeVdsBrokerCommand(ConnectStoragePoolVDSCommand.java:32) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source) [:1.8.0_191]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
        at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) [:1.8.0_191]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
        at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
        at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:]
        at org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation.execute(RefreshPoolSingleAsyncOperation.java:59) [bll.jar:]
        at org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:43) [utils.jar:]
        at org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:32) [utils.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:117) [utils.jar:]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191]
        at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:]