This is a multi-part message in MIME format. --------------D32DAC9929FA84F802C5B88B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in OST for basic disk operation. from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. Also I don't think the patch that is reported as the cause was actually the cause. I greped the image from the log: http://pastebin.test.redhat.com/520150 * Link to suspected patches: * https://gerrit.ovirt.org/#/c/82293/2 * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/ (Relevant) error snippet from the log: <error> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] </error> * --------------D32DAC9929FA84F802C5B88B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi, <br> </p> <p>We had a failure in OST for basic disk operation. <br> </p> <p>from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. <br> </p> <p>However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. <br> </p> <p>Also I don't think the patch that is reported as the cause was actually the cause. <br> </p> <p>I greped the image from the log: <a class="moz-txt-link-freetext" href="http://pastebin.test.redhat.com/520150">http://pastebin.test.redhat.com/520150</a><br> </p> <pre wrap=""> <b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: </span></p></b><pre wrap=""><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82293/2">https://gerrit.ovirt.org/#/c/82293/2</a></pre><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-c825-6405-6339-e584400f8792"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/</a></span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> </b></pre> </body> </html> --------------D32DAC9929FA84F802C5B88B--