<div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Sun, May 28, 2017 at 6:48 PM Benny Zlotnik <<a href="mailto:bzlotnik@redhat.com">bzlotnik@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Looking into this. Looks like all LSM tasks finished but the engine memory lock hasn't been released yet.</div></blockquote><div><br></div><div>Most likely the test or the sdk are not waiting correctly for the completion of the live merge.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 28, 2017 at 5:17 PM, Dafna Ron <span dir="ltr"><<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b">
</b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Hi All,
</span></b></p><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></b></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We are seeing the same failure</span></b><b><span style="font-weight:normal"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> sporadically</span></span></b><b><span style="font-weight:normal"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> in </span></span></b><span style="font-weight:normal"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">three</span></span><b><span style="font-weight:normal"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> different places (hotunplug disk).
</span></span></b></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b><span style="font-weight:normal"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We think this is a race condition which was introduced to OST with the the live storage migration test (lock on the disk is preventing the hotunplug).
</span></span></b></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Test failed: [ 004_basic_sanity.hotunplug_disk] </span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to suspected patches: <a class="m_-5643818663290888279m_-8269154987181321134moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/77083" target="_blank">https://gerrit.ovirt.org/#/c/77083</a></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job: <a class="m_-5643818663290888279m_-8269154987181321134moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881" target="_blank">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881</a></span></p>
</b><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b">
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs: <a class="m_-5643818663290888279m_-8269154987181321134moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/" target="_blank">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6881/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/</a></span></p>
<br>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Error snippet from the log: </span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><error></span></p>
</b><p></p>
<p><br>
</p>
<h3 style="box-sizing:border-box;color:rgb(51,51,51);font-family:Helvetica,Arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Error
Message</h3>
<pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">status: 409
reason: Conflict
detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.</pre>
<h3 style="box-sizing:border-box;color:rgb(51,51,51);font-family:Helvetica,Arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Stacktrace</h3>
<pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 68, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 498, in hotunplug_disk
disk.deactivate()
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31839, in deactivate
headers={"Correlation-Id":correlation_id}
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
persistent_auth=self.__persistent_auth
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
persistent_auth)
File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 409
reason: Conflict
detail: Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.
from engine log:
2017-05-28 09:05:32,891-04 INFO [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'
2017-05-28 09:05:32,891-04 WARN [org.ovirt.engine.core.bll.storage.disk.HotUnPlugDiskFromVmCommand] (default task-10) [6d47ba60-4491-48ab-bb1c-fb0cdbeb76b0] Validation of action 'HotUnPlugDiskFromVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__HOT_UNPLUG,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED,$DiskName vm0_disk0
2017-05-28 09:05:32,895-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-10) [] Operation Failed: [Cannot hot unplug Virtual Disk. Disk vm0_disk0 is being moved or copied.]
2017-05-28 09:05:33,323-04 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand' successfully. </pre></div></blockquote></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">2017-05-28 09:05:33,324-04 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (DefaultQuartzScheduler8) [99f56ceb-3c13-423f-b015-71222a8fec01] Lock freed to object 'EngineLock:{exclusiveLocks='[9e7d8b37-2afb-4012-9bc9-8cc277ac9cdd=<DISK, ACTION_TYPE_FAILED_DISK_IS_BEING_MIGRATED$DiskName vm0_disk0>]', sharedLocks='[461224f5-a2af-4be2-ba1b-3563a88f398c=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]'}' </pre></div></blockquote></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">
</pre>
<p><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b">
</b></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></error></span></b></p><b style="font-weight:normal" id="m_-5643818663290888279m_-8269154987181321134docs-internal-guid-582c7f51-4f4d-bb80-5302-dcf7903d7f8b">
<br>
</b><p></p>
</div>
</blockquote></div><br></div>
_______________________________________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></blockquote></div></div>