On Thu, Feb 28, 2019 at 12:01 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > Hi, > > The test failed due to not finding master storage domain > I am seeing a build of engine that passed so its possible that this is > related to the random failures we have been having on other tests but since > the change reported is storage related, can you please review the logs to > see if anything happened to the master domain while trying to resize? > First the patch cannot affect the storage domain, this is just moving of constant from one module to another, and verified by vdsm tests. If the patch was broken we would fail with a NameError in vdsm, accessing non-exisitng variable. I think it is a flaky test, since I ran many successful OST with this patch. These run include the patch: - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4173/ - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4172/ - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4171/ There may be real issue behind this, but it is not related to the patch. I think this is related to not deactivating lvs when putting storage domain in maintenance. We have this bug for the issue: https://bugzilla.redhat.com/1163890 The issue is probably random since storage domain monitor is refreshed every 5 minutes, so the system correct itself automatically. It probably fails if the test happened to start at the wrong time, timing out before the system refreshed the storage. If you see more of this issue, please file a new bug and we will handle it when we have time. In the meantime you can disable the test since it is not reliable. Nir > CQ reported this change as root cause: > https://gerrit.ovirt.org/#/c/98015/6 - blockSD: Move legacy block size > constants to sd > > build failure: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13175/ > > logs: > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13175/arti... > > errors in log: > > 2019-02-27 17:35:02,222-05 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] EVENT_ID: > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM la > go-basic-suite-master-host-1 command ConnectStoragePoolVDS failed: Cannot > find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, > msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' > 2019-02-27 17:35:02,222-05 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command > 'org.ovirt.engine.core.vdsbroker.vdsbroker. > ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn > [status=Status [code=304, message=Cannot find master domain: > u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, > msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5 > a9']]' > 2019-02-27 17:35:02,222-05 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] HostName = > lago-basic-suite-master-host-1 > 2019-02-27 17:35:02,223-05 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Command > 'ConnectStoragePoolVDSCommand(HostName = lago-basic-suite-master-host-1, > ConnectStoragePoolVDSCommandParameters:{hostId='1ac0eda2-0980-40df-b029-b262680696a4', > vdsId='1ac0eda2-0980-40df-b029-b262680696a4', > storagePoolId='1c79b83e-32d6-43e0-bc61-2955e42c3a58', masterVersion='1'})' > execution failed: IRSGenericException: IRSErrorException: > IRSNoMasterDomainException: Cannot find master domain: > u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, > msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' > 2019-02-27 17:35:02,223-05 DEBUG > [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-23) [558e4473] Exception: > org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: > IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot > find master domain: u'spUUID=1c79b83e-32d6-43e0-bc61-2955e42c3a58, > msdUUID=7a7e1d7b-08aa-49f6-bdf2-4be64260b5a9' > at > org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:50) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedConnectProxyReturnValue(ConnectStoragePoolVDSCommand.java:48) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedProxyReturnValue(ConnectStoragePoolVDSCommand.java:36) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.executeVdsBrokerCommand(ConnectStoragePoolVDSCommand.java:32) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) > [vdsbroker.jar:] > at > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) > [dal.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:396) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown > Source) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor252.invoke(Unknown Source) > [:1.8.0_191] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_191] > at java.lang.reflect.Method.invoke(Method.java:498) > [rt.jar:1.8.0_191] > at > org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) > [common.jar:] > at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) > [:1.8.0_191] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_191] > at java.lang.reflect.Method.invoke(Method.java:498) > [rt.jar:1.8.0_191] > at > org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) > [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] > at > org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown > Source) [vdsbroker.jar:] > at > org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation.execute(RefreshPoolSingleAsyncOperation.java:59) > [bll.jar:] > at > org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:43) > [utils.jar:] > at > org.ovirt.engine.core.utils.SynchronizeNumberOfAsyncOperations$AsyncOpThread.call(SynchronizeNumberOfAsyncOperations.java:32) > [utils.jar:] > at > org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:117) > [utils.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [rt.jar:1.8.0_191] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_191] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [rt.jar:1.8.0_191] > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191] > at > org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) > [javax.enterprise.concurrent-1.0.jar:] > >