We should probably lock both groups in VdsEventListener#processStorageOnVdsInactive or in RemoveVdsCommand Ahmad, please evaluate and adjust On Wed, Mar 20, 2019 at 11:25 AM Eitan Raviv <eraviv@redhat.com> wrote:
At least for the network suite these failures are due to the fact that DisconnectStoragePoolVDSCommand is not finished and a change_cluster request is issued by the test but revoked by engine which fails the test setup.
Ahmed, Can you please have a look at the below engine log [2]- currently change_cluster requests from the test are being revoked by engine. I believe this is because the locking group of DisconnectStoragePoolVDSCommand was moved to VDS in patch [1] Is this a desired outcome and should the tests be changed to reflect it (wait for which status notification from engine?)?
Thanks
P.S. please note that due to this patch LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS has only one consumer, which means it does not lock against anything. Is this a desired outcome of the patch? [1] https://gerrit.ovirt.org/#/c/98405 [2] Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap Error Message
test setup failure Stacktrace
default_data_center = <ovirtlib.datacenterlib.DataCenter object at 0x7fbe88aa8790> host_1_up = <ovirtlib.hostlib.Host object at 0x7fbe8404a810> cluster_0 = <ovirtlib.clusterlib.Cluster object at 0x7fbe77f5a050>
@pytest.fixture(scope='module') def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0): current_cluster = host_1_up.get_cluster()
host_1_up.change_cluster(cluster_0)
network-suite-master/tests/test_mac_pools.py:49
2019-03-17 07:16:40,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-network-suite-master-host-1, DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42', storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}), log id: 44ad4438
2019-03-17 07:16:43,635-04 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]', sharedLocks=''}' 2019-03-17 07:16:43,635-04 WARN [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action 'ChangeVDSCluster' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED 2019-03-17 07:16:43,636-04 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction, params: [ChangeVDSCluster, ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24', user='null', commandType='Unknown'}], timeElapsed: 18ms 2019-03-17 07:16:43,637-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [Cannot edit Host. Related operation is currently in progress. Please try again later.]
2019-03-17 07:16:44,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH, DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438
On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri <eedri@redhat.com> wrote:
Not sure if all the same issue, but seems to failing around the same time:
  ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57 min  integ-tests   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr - #1045 51 min  integ-tests   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec  integ-tests   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min  integ-tests   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 hr - #873 1 hr 36 min  integ-tests   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr - #1149 1 hr 26 min  integ-tests   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr - #729 1 hr 54 min  integ-tests   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - #929 45 min  integ-tests   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 hr - #190 45 min  integ-tests
Links to jobs can be found here: https://jenkins.ovirt.org/
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/I75ZI4LYD3IKZC...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UXLFIW2KK4JWJO...