At least for the network suite these failures are due to the fact that DisconnectStoragePoolVDSCommand is not finished and a change_cluster request is issued by the test but revoked by engine which fails the test setup.

Ahmed,

Can you please have a look at the below engine log [2]- currently change_cluster requests from the test are being revoked by engine. I believe this is because the locking group of DisconnectStoragePoolVDSCommand was moved to VDS in patch [1]

Is this a desired outcome and should the tests be changed to reflect it (wait for which status notification from engine?)?

Thanks

P.S. please note that due to this patch LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS has only one consumer, which means it does not lock against anything. Is this a desired outcome of the patch?

[1] https://gerrit.ovirt.org/#/c/98405

[2] Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap

Error Message

test setup failure

Stacktrace

default_data_center = <ovirtlib.datacenterlib.DataCenter object at 0x7fbe88aa8790>
host_1_up = <ovirtlib.hostlib.Host object at 0x7fbe8404a810>
cluster_0 = <ovirtlib.clusterlib.Cluster object at 0x7fbe77f5a050>

   @pytest.fixture(scope='module')
   def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0):
       current_cluster = host_1_up.get_cluster()

>       host_1_up.change_cluster(cluster_0)

network-suite-master/tests/test_mac_pools.py:49

2019-03-17 07:16:40,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-network-suite-master-host-1, DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42', storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}), log id: 44ad4438

2019-03-17 07:16:43,635-04 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]', sharedLocks=''}'
2019-03-17 07:16:43,635-04 WARN [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action 'ChangeVDSCluster' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
2019-03-17 07:16:43,636-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction, params: [ChangeVDSCluster, ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24', user='null', commandType='Unknown'}], timeElapsed: 18ms
2019-03-17 07:16:43,637-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [Cannot edit Host. Related operation is currently in progress. Please try again later.]

2019-03-17 07:16:44,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH, DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438

On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri <eedri@redhat.com> wrote:

Not sure if all the same issue, but seems to failing around the same time:

ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57 min integ-tests
ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr - #1045 51 min integ-tests
ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec integ-tests
ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min integ-tests
ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 hr - #873 1 hr 36 min integ-tests
ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr - #1149 1 hr 26 min integ-tests
ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr - #729 1 hr 54 min integ-tests
ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929 45 min integ-tests
ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 hr - #190 45 min integ-tests

Links to jobs can be found here: https://jenkins.ovirt.org/

--
Eyal edri

MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D

Red Hat EMEA
TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/I75ZI4LYD3IKZCSZEHNU7IR6K7DDRU3K/