[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

At least for the network suite these failures are due to the fact that DisconnectStoragePoolVDSCommand is not finished and a change_cluster request is issued by the test but revoked by engine which fails the test setup. Ahmed, Can you please have a look at the below engine log [2]- currently change_cluster requests from the test are being revoked by engine. I believe this is because the locking group of DisconnectStoragePoolVDSCommand was moved to VDS in patch [1] Is this a desired outcome and should the tests be changed to reflect it (wait for which status notification from engine?)? Thanks P.S. please note that due to this patch LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS has only one consumer, which means it does not lock against anything. Is this a desired outcome of the patch? [1] https://gerrit.ovirt.org/#/c/98405 [2] Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap Error Message test setup failure Stacktrace default_data_center = <ovirtlib.datacenterlib.DataCenter object at 0x7fbe88aa8790> host_1_up = <ovirtlib.hostlib.Host object at 0x7fbe8404a810> cluster_0 = <ovirtlib.clusterlib.Cluster object at 0x7fbe77f5a050> @pytest.fixture(scope='module') def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0): current_cluster = host_1_up.get_cluster() network-suite-master/tests/test_mac_pools.py:49 2019-03-17 07:16:40,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-network-suite-master-host-1, DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42', storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}), log id: 44ad4438 2019-03-17 07:16:43,635-04 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]', sharedLocks=''}' 2019-03-17 07:16:43,635-04 WARN [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action 'ChangeVDSCluster' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED 2019-03-17 07:16:43,636-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction, params: [ChangeVDSCluster, ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24', user='null', commandType='Unknown'}], timeElapsed: 18ms 2019-03-17 07:16:43,637-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [Cannot edit Host. Related operation is currently in progress. Please try again later.] 2019-03-17 07:16:44,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH, DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438 On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: