On Sun, May 28, 2017 at 6:48 PM Benny Zlotnik <bzlotnik@redhat.com> wrote:
Looking into this. Looks like all LSM tasks finished but the engine memory lock hasn't been released yet.
Most likely the test or the sdk are not waiting correctly for the completion of the live merge.
On Sun, May 28, 2017 at 5:17 PM, Dafna Ron <dron@redhat.com> wrote:
*Hi All, *
* We are seeing the same failure sporadically in three different places (hotunplug disk). We think this is a race condition which was introduced to OST with the the live storage migration test (lock on the disk is preventing the hotunplug). Test failed: [ 004_basic_sanity.hotunplug_disk] Link to suspected patches: https://gerrit.ovirt.org/#/c/77083 <https://gerrit.ovirt.org/#/c/77083> Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881 <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881> *
* Link to all logs: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifa... <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/> Error snippet from the log: <error> *
Error Message
status: 409 reason: Conflict detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.
Stacktrace
Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 498, in hotunplug_disk disk.deactivate() File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31839, in deactivate headers={"Correlation-Id":correlation_id} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.
from engine log: 2017-05-28 09:05:32,891-04 INFO [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}' 2017-05-28 09:05:32,891-04 WARN [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Validation of action 'HotUnPlugDiskFromVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__HOT_UNPLUG,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED,$DiskName vm0_disk0 2017-05-28 09:05:32,895-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-10) [] Operation Failed: [Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.] 2017-05-28 09:05:33,323-04 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand' successfully.
2017-05-28 09:05:33,324-04 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Lock freed to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED$DiskName vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]'}'
*</error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel