[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 28-05-2017 ] [ 004_basic_sanity.hotunplug_disk ]

[Dafna Ron]

**

*Hi All, *

*

**

*We are seeing the same failure**sporadically**in *three*different
places (hotunplug disk). *

*We think this is a race condition which was introduced to OST with the
the live storage migration test (lock on the disk is preventing the
hotunplug). *

Test failed: [ 004_basic_sanity.hotunplug_disk]

Link to suspected patches: https://gerrit.ovirt.org/#/c/77083

Link to Job:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881

**

Link to all logs:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/


Error snippet from the log:

<error>

*


      Error Message

status: 409
reason: Conflict
detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.


      Stacktrace

Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test
    test()
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper
    return func(get_test_prefix(), *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper
    return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
  File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 498, in hotunplug_disk
    disk.deactivate()
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31839, in deactivate
    headers={"Correlation-Id":correlation_id}
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
    persistent_auth=self.__persistent_auth
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
    persistent_auth)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
    raise errors.RequestError(response_code, response_reason, response_body)
RequestError: 
status: 409
reason: Conflict
detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.


from engine log: 
2017-05-28 09:05:32,891-04 INFO  [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'
2017-05-28 09:05:32,891-04 WARN  [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Validation of action 'HotUnPlugDiskFromVm' failed for user admin at internal-authz. Reasons: VAR__ACTION__HOT_UNPLUG,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED,$DiskName vm0_disk0
2017-05-28 09:05:32,895-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-10) [] Operation Failed: [Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.]
2017-05-28 09:05:33,323-04 INFO  [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand' successfully.
2017-05-28 09:05:33,324-04 INFO  [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Lock freed to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED$DiskName vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]'}'

**

*</error>*

*
*

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170528/c0f5fa28/attachment-0001.html>