6:05 a.m.
This is a multi-part message in MIME format.
--------------D32DAC9929FA84F802C5B88B
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
We had a failure in OST for basic disk operation.
from what I can see in the log, there is an issue deleting a snapshot
image and the issue is related to the live storage migration.
However, a second pair of eyes would help since I can see engine
reporting the error on failed delete while vdsm was not actually
reporting a failed delete, at least not as ERROR.
Also I don't think the patch that is reported as the cause was actually
the cause.
I greped the image from the log: http://pastebin.test.redhat.com/520150
*
Link to suspected patches:
*
https://gerrit.ovirt.org/#/c/82293/2
*
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/
(Relevant) error snippet from the log:
<error>
2017-09-27 16:51:02,760-04 ERROR
[org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-6)
[8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not
removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe]
</error>
*
--------------D32DAC9929FA84F802C5B88B
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in OST for basic disk operation. <br>
</p>
<p>from what I can see in the log, there is an issue deleting a
snapshot image and the issue is related to the live storage
migration. <br>
</p>
<p>However, a second pair of eyes would help since I can see engine
reporting the error on failed delete while vdsm was not actually
reporting a failed delete, at least not as ERROR. <br>
</p>
<p>Also I don't think the patch that is reported as the cause was
actually the cause. <br>
</p>
<p>I greped the image from the log:
<a class="moz-txt-link-freetext"
href="http://pastebin.test.redhat.com/520150">http://pastebi...
</p>
<pre wrap="">
<b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: </span></p></b><pre wrap=""><a
class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/82293/2">https://gerrit.o...
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-6)
[8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed:
[be38d24d-33b9-43f0-
9ab4-15f0334e2ebe]
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></pre>
</body>
</html>
--------------D32DAC9929FA84F802C5B88B--
6:08 a.m.
This is a multi-part message in MIME format.
--------------17AEB594933D418AE946AB63
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
just to add that we had 2 other failures for basic disk operation after
the one below:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2912/
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2909/
and the tests started passing on:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2913/
On 09/28/2017 12:05 PM, Dafna Ron wrote:
--------------17AEB594933D418AE946AB63
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">just to add that we had 2 other
failures for basic disk operation after the one below: <br>
<br>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
and the tests started passing on: <br>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
<br>
On 09/28/2017 12:05 PM, Dafna Ron wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<p>Hi, <br>
</p>
<p>We had a failure in OST for basic disk operation. <br>
</p>
<p>from what I can see in the log, there is an issue deleting a
snapshot image and the issue is related to the live storage
migration. <br>
</p>
<p>However, a second pair of eyes would help since I can see
engine reporting the error on failed delete while vdsm was not
actually reporting a failed delete, at least not as ERROR. <br>
</p>
<p>Also I don't think the patch that is reported as the cause was
actually the cause. <br>
</p>
<p>I greped the image from the log: <a
class="moz-txt-link-freetext"
href="http://pastebin.test.redhat.com/520150"
moz-do-not-send="true">http://pastebin.test.redhat.com/52015...
</p>
<pre wrap="">
<b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: </span></p></b><pre wrap=""><a
class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/82293/2"
moz-do-not-send="true">https://gerrit.ovirt.org/#/c/82293/2&...
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-ma...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-ma...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-6)
[8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed:
[be38d24d-33b9-43f0-
9ab4-15f0334e2ebe]
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Devel mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Devel@ovirt.org">Devel@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/devel">http://...
</blockquote>
<p><br>
</p>
</body>
</html>
--------------17AEB594933D418AE946AB63--
7:46 a.m.
This is a multi-part message in MIME format.
--------------969AC12EBD7819EE5C37AA1C
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Please note that 004_basic_sanity.disk_operations is still failing:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2941/
thanks,
Dafna
On 09/28/2017 12:05 PM, Dafna Ron wrote:
--------------969AC12EBD7819EE5C37AA1C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Please note that
004_basic_sanity.disk_operations is still failing: <br>
<br>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
thanks, <br>
Dafna<br>
<br>
<br>
On 09/28/2017 12:05 PM, Dafna Ron wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<p>Hi, <br>
</p>
<p>We had a failure in OST for basic disk operation. <br>
</p>
<p>from what I can see in the log, there is an issue deleting a
snapshot image and the issue is related to the live storage
migration. <br>
</p>
<p>However, a second pair of eyes would help since I can see
engine reporting the error on failed delete while vdsm was not
actually reporting a failed delete, at least not as ERROR. <br>
</p>
<p>Also I don't think the patch that is reported as the cause was
actually the cause. <br>
</p>
<p>I greped the image from the log: <a
class="moz-txt-link-freetext"
href="http://pastebin.test.redhat.com/520150"
moz-do-not-send="true">http://pastebin.test.redhat.com/52015...
</p>
<pre wrap="">
<b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p
dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: </span></p></b><pre wrap=""><a
class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/82293/2"
moz-do-not-send="true">https://gerrit.ovirt.org/#/c/82293/2&...
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-ma...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-ma...
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-6)
[8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed:
[be38d24d-33b9-43f0-
9ab4-15f0334e2ebe]
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Infra mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Infra@ovirt.org">Infra@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/infra">http://...
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>
--------------969AC12EBD7819EE5C37AA1C--
2745
days inactive
2746
days old
2 comments
1 participants
participants (1)
-
Dafna Ron