[ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 004_basic_sanity.disk_operations ]
This is a multi-part message in MIME format. --------------D32DAC9929FA84F802C5B88B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in OST for basic disk operation. from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. Also I don't think the patch that is reported as the cause was actually the cause. I greped the image from the log: http://pastebin.test.redhat.com/520150 * Link to suspected patches: * https://gerrit.ovirt.org/#/c/82293/2 * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/ (Relevant) error snippet from the log: <error> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] </error> * --------------D32DAC9929FA84F802C5B88B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi, <br> </p> <p>We had a failure in OST for basic disk operation. <br> </p> <p>from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. <br> </p> <p>However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. <br> </p> <p>Also I don't think the patch that is reported as the cause was actually the cause. <br> </p> <p>I greped the image from the log: <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520150">http://pastebin.test.redhat.com/520150</a><br> </p> <pre wrap=""> <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: </span></p></b><pre wrap=""><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82293/2">https://gerrit.ovirt.org/#/c/82293/2</a></pre><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> </b></pre> </body> </html> --------------D32DAC9929FA84F802C5B88B--
This is a multi-part message in MIME format. --------------17AEB594933D418AE946AB63 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit just to add that we had 2 other failures for basic disk operation after the one below: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2912/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2909/ and the tests started passing on: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2913/ On 09/28/2017 12:05 PM, Dafna Ron wrote:
Hi,
We had a failure in OST for basic disk operation.
from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration.
However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR.
Also I don't think the patch that is reported as the cause was actually the cause.
I greped the image from the log: http://pastebin.test.redhat.com/520150
*
Link to suspected patches:
* https://gerrit.ovirt.org/#/c/82293/2 *
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/
(Relevant) error snippet from the log:
<error>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe]
</error>
*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--------------17AEB594933D418AE946AB63 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">just to add that we had 2 other failures for basic disk operation after the one below: <br> <br> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2912/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2912/</a><br> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2909/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2909/</a><br> <br> and the tests started passing on: <br> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2913/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2913/</a><br> <br> <br> On 09/28/2017 12:05 PM, Dafna Ron wrote:<br> </div> <blockquote type="cite" cite="mid:ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com"> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <p>Hi, <br> </p> <p>We had a failure in OST for basic disk operation. <br> </p> <p>from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. <br> </p> <p>However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. <br> </p> <p>Also I don't think the patch that is reported as the cause was actually the cause. <br> </p> <p>I greped the image from the log: <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520150" moz-do-not-send="true">http://pastebin.test.redhat.com/520150</a><br> </p> <pre wrap=""> <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: </span></p></b><pre wrap=""><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82293/2" moz-do-not-send="true">https://gerrit.ovirt.org/#/c/82293/2</a></pre><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> </b></pre> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a></pre> </blockquote> <p><br> </p> </body> </html> --------------17AEB594933D418AE946AB63--
This is a multi-part message in MIME format. --------------969AC12EBD7819EE5C37AA1C Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Please note that 004_basic_sanity.disk_operations is still failing: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2941/ thanks, Dafna On 09/28/2017 12:05 PM, Dafna Ron wrote:
Hi,
We had a failure in OST for basic disk operation.
from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration.
However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR.
Also I don't think the patch that is reported as the cause was actually the cause.
I greped the image from the log: http://pastebin.test.redhat.com/520150
*
Link to suspected patches:
* https://gerrit.ovirt.org/#/c/82293/2 *
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/
(Relevant) error snippet from the log:
<error>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe]
</error>
*
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--------------969AC12EBD7819EE5C37AA1C Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">Please note that 004_basic_sanity.disk_operations is still failing: <br> <br> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2941/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2941/</a><br> <br> thanks, <br> Dafna<br> <br> <br> On 09/28/2017 12:05 PM, Dafna Ron wrote:<br> </div> <blockquote type="cite" cite="mid:ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com"> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <p>Hi, <br> </p> <p>We had a failure in OST for basic disk operation. <br> </p> <p>from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. <br> </p> <p>However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. <br> </p> <p>Also I don't think the patch that is reported as the cause was actually the cause. <br> </p> <p>I greped the image from the log: <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520150" moz-do-not-send="true">http://pastebin.test.redhat.com/520150</a><br> </p> <pre wrap=""> <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: </span></p></b><pre wrap=""><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82293/2" moz-do-not-send="true">https://gerrit.ovirt.org/#/c/82293/2</a></pre><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/" moz-do-not-send="true">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> </b></pre> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Infra mailing list <a class="moz-txt-link-abbreviated" href="mailto:Infra@ovirt.org">Infra@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/infra">http://lists.ovirt.org/mailman/listinfo/infra</a> </pre> </blockquote> <p><br> </p> </body> </html> --------------969AC12EBD7819EE5C37AA1C--
participants (1)
-
Dafna Ron