From dron at redhat.com Thu Sep 28 11:05:54 2017 Content-Type: multipart/mixed; boundary="===============7512301075010782464==" MIME-Version: 1.0 From: Dafna Ron To: devel at ovirt.org Subject: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 004_basic_sanity.disk_operations ] Date: Thu, 28 Sep 2017 12:05:51 +0100 Message-ID: --===============7512301075010782464== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------D32DAC9929FA84F802C5B88B Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in OST for basic disk operation. from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration. However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR. Also I don't think the patch that is reported as the cause was actually the cause. I greped the image from the log: http://pastebin.test.redhat.com/520150 * Link to suspected patches: * https://gerrit.ovirt.org/#/c/82293/2 * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifact/ (Relevant) error snippet from the log: 2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] * --------------D32DAC9929FA84F802C5B88B Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 7bit

Hi,

We had a failure in OST for basic disk operation.

from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration.

However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR.

Also I don't think the patch that is reported as the cause was actually the cause.

I greped the image from the log: http://pastebin.test.redhat.com/520150


Link to suspected patches: ht=
tps://gerrit.ovirt.org/#/c/82293/2

Link to Job=
: http://jenkins.ovirt.org/job/ovirt-ma=
ster_change-queue-tester/2906
Link to all=
 logs: http://jenkins.ovirt.o=
rg/job/ovirt-master_change-queue-tester/2906/artifact/
(Relevant) =
error snippet from the log: 
<error>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageChe=
ckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-=
3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d2=
4d-33b9-43f0-
9ab4-15f0334e2ebe]


</error&=
gt;

--------------D32DAC9929FA84F802C5B88B-- --===============7512301075010782464== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1EMzJEQUM5OTI5RkE4NEY4MDJDNUI4OEIKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCkhpLAoKV2UgaGFkIGEg ZmFpbHVyZSBpbiBPU1QgZm9yIGJhc2ljIGRpc2sgb3BlcmF0aW9uLgoKZnJvbSB3aGF0IEkgY2Fu IHNlZSBpbiB0aGUgbG9nLCB0aGVyZSBpcyBhbiBpc3N1ZSBkZWxldGluZyBhIHNuYXBzaG90Cmlt YWdlIGFuZCB0aGUgaXNzdWUgaXMgcmVsYXRlZCB0byB0aGUgbGl2ZSBzdG9yYWdlIG1pZ3JhdGlv bi4KCkhvd2V2ZXIsIGEgc2Vjb25kIHBhaXIgb2YgZXllcyB3b3VsZCBoZWxwIHNpbmNlIEkgY2Fu IHNlZSBlbmdpbmUKcmVwb3J0aW5nIHRoZSBlcnJvciBvbiBmYWlsZWQgZGVsZXRlIHdoaWxlIHZk c20gd2FzIG5vdCBhY3R1YWxseQpyZXBvcnRpbmcgYSBmYWlsZWQgZGVsZXRlLCBhdCBsZWFzdCBu b3QgYXMgRVJST1IuCgpBbHNvIEkgZG9uJ3QgdGhpbmsgdGhlIHBhdGNoIHRoYXQgaXMgcmVwb3J0 ZWQgYXMgdGhlIGNhdXNlIHdhcyBhY3R1YWxseQp0aGUgY2F1c2UuCgpJIGdyZXBlZCB0aGUgaW1h Z2UgZnJvbSB0aGUgbG9nOiBodHRwOi8vcGFzdGViaW4udGVzdC5yZWRoYXQuY29tLzUyMDE1MAoK CioKCkxpbmsgdG8gc3VzcGVjdGVkIHBhdGNoZXM6CgoqCgpodHRwczovL2dlcnJpdC5vdmlydC5v cmcvIy9jLzgyMjkzLzIKCioKCkxpbmsgdG8gSm9iOgpodHRwOi8vamVua2lucy5vdmlydC5vcmcv am9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MDYKCkxpbmsgdG8gYWxsIGxv Z3M6Cmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1 ZS10ZXN0ZXIvMjkwNi9hcnRpZmFjdC8KCihSZWxldmFudCkgZXJyb3Igc25pcHBldCBmcm9tIHRo ZSBsb2c6Cgo8ZXJyb3I+CgoyMDE3LTA5LTI3IDE2OjUxOjAyLDc2MC0wNCBFUlJPUgpbb3JnLm92 aXJ0LmVuZ2luZS5jb3JlLmJsbC5EZXN0cm95SW1hZ2VDaGVja0NvbW1hbmRdCihFRS1NYW5hZ2Vk VGhyZWFkRmFjdG9yeS1jb21tYW5kQ29vcmRpbmF0b3ItVGhyZWFkLTYpCls4OTgxYTBmNC0zYzAy LTRhMmQtYjZiYS1mMjEyMTRiMGViZDRdIFRoZSBmb2xsb3dpbmcgaW1hZ2VzIHdlcmUgbm90CnJl bW92ZWQ6IFtiZTM4ZDI0ZC0zM2I5LTQzZjAtIDlhYjQtMTVmMDMzNGUyZWJlXQoKPC9lcnJvcj4K CioKCgotLS0tLS0tLS0tLS0tLUQzMkRBQzk5MjlGQTg0RjgwMkM1Qjg4QgpDb250ZW50LVR5cGU6 IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3Yml0 Cgo8aHRtbD4KICA8aGVhZD4KCiAgICA8bWV0YSBodHRwLWVxdWl2PSJjb250ZW50LXR5cGUiIGNv bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRleHQ9 IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPHA+SGksIDxicj4KICAgIDwvcD4KICAg IDxwPldlIGhhZCBhIGZhaWx1cmUgaW4gT1NUIGZvciBiYXNpYyBkaXNrIG9wZXJhdGlvbi4gPGJy PgogICAgPC9wPgogICAgPHA+ZnJvbSB3aGF0IEkgY2FuIHNlZSBpbiB0aGUgbG9nLCB0aGVyZSBp cyBhbiBpc3N1ZSBkZWxldGluZyBhCiAgICAgIHNuYXBzaG90IGltYWdlIGFuZCB0aGUgaXNzdWUg aXMgcmVsYXRlZCB0byB0aGUgbGl2ZSBzdG9yYWdlCiAgICAgIG1pZ3JhdGlvbi4gPGJyPgogICAg PC9wPgogICAgPHA+SG93ZXZlciwgYSBzZWNvbmQgcGFpciBvZiBleWVzIHdvdWxkIGhlbHAgc2lu Y2UgSSBjYW4gc2VlIGVuZ2luZQogICAgICByZXBvcnRpbmcgdGhlIGVycm9yIG9uIGZhaWxlZCBk ZWxldGUgd2hpbGUgdmRzbSB3YXMgbm90IGFjdHVhbGx5CiAgICAgIHJlcG9ydGluZyBhIGZhaWxl ZCBkZWxldGUsIGF0IGxlYXN0IG5vdCBhcyBFUlJPUi4gPGJyPgogICAgPC9wPgogICAgPHA+QWxz byBJIGRvbid0IHRoaW5rIHRoZSBwYXRjaCB0aGF0IGlzIHJlcG9ydGVkIGFzIHRoZSBjYXVzZSB3 YXMKICAgICAgYWN0dWFsbHkgdGhlIGNhdXNlLiA8YnI+CiAgICA8L3A+CiAgICA8cD5JIGdyZXBl ZCB0aGUgaW1hZ2UgZnJvbSB0aGUgbG9nOgogICAgICA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZy ZWV0ZXh0IiBocmVmPSJodHRwOi8vcGFzdGViaW4udGVzdC5yZWRoYXQuY29tLzUyMDE1MCI+aHR0 cDovL3Bhc3RlYmluLnRlc3QucmVkaGF0LmNvbS81MjAxNTA8L2E+PGJyPgogICAgPC9wPgogICAg PHByZSB3cmFwPSIiPgoKPGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbDsiIGlkPSJkb2NzLWlu dGVybmFsLWd1aWQtNTg1OWI3YTEtYzgyNS02NDA1LTYzMzktZTU4NDQwMGY4NzkyIj48cCBkaXI9 Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRv bTowcHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29s b3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtm b250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25l O3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwOyI+TGluayB0byBz dXNwZWN0ZWQgcGF0Y2hlczogPC9zcGFuPjwvcD48L2I+PHByZSB3cmFwPSIiPjxhIGNsYXNzPSJt b3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2Mv ODIyOTMvMiI+aHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy84MjI5My8yPC9hPjwvcHJlPjxi IHN0eWxlPSJmb250LXdlaWdodDpub3JtYWw7IiBpZD0iZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTli N2ExLWM4MjUtNjQwNS02MzM5LWU1ODQ0MDBmODc5MiI+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGlu ZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tn cm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFs O2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246 YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij5MaW5rIHRvIEpvYjogPGEgY2xhc3M9Im1v ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9v dmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTA2Ij5odHRwOi8vamVua2lucy5vdmly dC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MDY8L2E+PC9zcGFu PjwvcD4KPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0 O21hcmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFt aWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250 LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRl Y29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3Jh cDsiPkxpbmsgdG8gYWxsIGxvZ3M6IDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhy ZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1 ZS10ZXN0ZXIvMjkwNi9hcnRpZmFjdC8iPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3Zp cnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjkwNi9hcnRpZmFjdC88L2E+PC9zcGFuPjwv cD4KPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21h cmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5 OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdl aWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29y YXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsi PihSZWxldmFudCkgZXJyb3Igc25pcHBldCBmcm9tIHRoZSBsb2c6IDwvc3Bhbj48L3A+PHAgZGly PSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0 b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2Nv bG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7 Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9u ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPiZsdDtlcnJv ciZndDs8L3NwYW4+PC9wPgoyMDE3LTA5LTI3IDE2OjUxOjAyLDc2MC0wNCBFUlJPUiBbb3JnLm92 aXJ0LmVuZ2luZS5jb3JlLmJsbC5EZXN0cm95SW1hZ2VDaGVja0NvbW1hbmRdIChFRS1NYW5hZ2Vk VGhyZWFkRmFjdG9yeS1jb21tYW5kQ29vcmRpbmF0b3ItVGhyZWFkLTYpIFs4OTgxYTBmNC0zYzAy LTRhMmQtYjZiYS1mMjEyMTRiMGViZDRdIFRoZSBmb2xsb3dpbmcgaW1hZ2VzIHdlcmUgbm90IHJl bW92ZWQ6IFtiZTM4ZDI0ZC0zM2I5LTQzZjAtCjlhYjQtMTVmMDMzNGUyZWJlXQoKCjxwIGRpcj0i bHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9t OjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xv cjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2Zv bnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7 dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij4mbHQ7L2Vycm9y Jmd0Ozwvc3Bhbj48L3A+Cgo8L2I+PC9wcmU+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0t LS0tLUQzMkRBQzk5MjlGQTg0RjgwMkM1Qjg4Qi0tCg== --===============7512301075010782464==-- From dron at redhat.com Thu Sep 28 11:08:46 2017 Content-Type: multipart/mixed; boundary="===============3402953656035597619==" MIME-Version: 1.0 From: Dafna Ron To: devel at ovirt.org Subject: Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 004_basic_sanity.disk_operations ] Date: Thu, 28 Sep 2017 12:08:44 +0100 Message-ID: <39b02837-7b17-614b-ee2a-25794fe3b61e@redhat.com> In-Reply-To: ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com --===============3402953656035597619== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------17AEB594933D418AE946AB63 Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit just to add that we had 2 other failures for basic disk operation after the one below: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2912/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2909/ and the tests started passing on: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2913/ On 09/28/2017 12:05 PM, Dafna Ron wrote: > > Hi, > > We had a failure in OST for basic disk operation. > > from what I can see in the log, there is an issue deleting a snapshot > image and the issue is related to the live storage migration. > > However, a second pair of eyes would help since I can see engine > reporting the error on failed delete while vdsm was not actually > reporting a failed delete, at least not as ERROR. > > Also I don't think the patch that is reported as the cause was > actually the cause. > > I greped the image from the log: http://pastebin.test.redhat.com/520150 > > * > > Link to suspected patches: > > * > https://gerrit.ovirt.org/#/c/82293/2 > * > > Link to Job: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906 > > Link to all logs: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifa= ct/ > > (Relevant) error snippet from the log: > > > > 2017-09-27 16:51:02,760-04 ERROR > [org.ovirt.engine.core.bll.DestroyImageCheckCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-6) > [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not > removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] > > > > * > > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel --------------17AEB594933D418AE946AB63 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 7bit

just to add that we had 2 other failures for basic disk operation after the one below:

http://jenkins.ovirt.org/job/ovirt-mas= ter_change-queue-tester/2912/
http://jenkins.ovirt.org/job/ovirt-mas= ter_change-queue-tester/2909/

and the tests started passing on:
http://jenkins.ovirt.org/job/ovirt-mas= ter_change-queue-tester/2913/

On 09/28/2017 12:05 PM, Dafna Ron wrote:

Hi,

We had a failure in OST for basic disk operation.

from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration.

However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR.

Also I don't think the patch that is reported as the cause was actually the cause.

I greped the image from the log: http://pastebin.test.redhat.com/520150
Link to suspected patches: https://gerrit.ovirt.org/#/c/82293/2

Link to Job=
: http://jenki=
ns.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all=
 logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artif=
act/
(Relevant) =
error snippet from the log: 
<error>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageChe=
ckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-=
3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d2=
4d-33b9-43f0-
9ab4-15f0334e2ebe]


</error&=
gt;
_______________________________________________
Devel mailing list
Dev=
el(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

--------------17AEB594933D418AE946AB63-- --===============3402953656035597619== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0xN0FFQjU5NDkzM0Q0MThBRTk0NkFCNjMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCmp1c3QgdG8gYWRkIHRo YXQgd2UgaGFkIDIgb3RoZXIgZmFpbHVyZXMgZm9yIGJhc2ljIGRpc2sgb3BlcmF0aW9uIGFmdGVy CnRoZSBvbmUgYmVsb3c6CgpodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rl cl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MTIvCmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Iv b3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjkwOS8KCmFuZCB0aGUgdGVzdHMgc3Rh cnRlZCBwYXNzaW5nIG9uOgpodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rl cl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MTMvCgoKT24gMDkvMjgvMjAxNyAxMjowNSBQTSwgRGFm bmEgUm9uIHdyb3RlOgo+Cj4gSGksCj4KPiBXZSBoYWQgYSBmYWlsdXJlIGluIE9TVCBmb3IgYmFz aWMgZGlzayBvcGVyYXRpb24uCj4KPiBmcm9tIHdoYXQgSSBjYW4gc2VlIGluIHRoZSBsb2csIHRo ZXJlIGlzIGFuIGlzc3VlIGRlbGV0aW5nIGEgc25hcHNob3QKPiBpbWFnZSBhbmQgdGhlIGlzc3Vl IGlzIHJlbGF0ZWQgdG8gdGhlIGxpdmUgc3RvcmFnZSBtaWdyYXRpb24uCj4KPiBIb3dldmVyLCBh IHNlY29uZCBwYWlyIG9mIGV5ZXMgd291bGQgaGVscCBzaW5jZSBJIGNhbiBzZWUgZW5naW5lCj4g cmVwb3J0aW5nIHRoZSBlcnJvciBvbiBmYWlsZWQgZGVsZXRlIHdoaWxlIHZkc20gd2FzIG5vdCBh Y3R1YWxseQo+IHJlcG9ydGluZyBhIGZhaWxlZCBkZWxldGUsIGF0IGxlYXN0IG5vdCBhcyBFUlJP Ui4KPgo+IEFsc28gSSBkb24ndCB0aGluayB0aGUgcGF0Y2ggdGhhdCBpcyByZXBvcnRlZCBhcyB0 aGUgY2F1c2Ugd2FzCj4gYWN0dWFsbHkgdGhlIGNhdXNlLgo+Cj4gSSBncmVwZWQgdGhlIGltYWdl IGZyb20gdGhlIGxvZzogaHR0cDovL3Bhc3RlYmluLnRlc3QucmVkaGF0LmNvbS81MjAxNTAKPgo+ ICoKPgo+IExpbmsgdG8gc3VzcGVjdGVkIHBhdGNoZXM6Cj4KPiAqCj4gaHR0cHM6Ly9nZXJyaXQu b3ZpcnQub3JnLyMvYy84MjI5My8yCj4gKgo+Cj4gTGluayB0byBKb2I6Cj4gaHR0cDovL2plbmtp bnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTA2Cj4K PiBMaW5rIHRvIGFsbCBsb2dzOgo+IGh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQt bWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjkwNi9hcnRpZmFjdC8KPgo+IChSZWxldmFudCkg ZXJyb3Igc25pcHBldCBmcm9tIHRoZSBsb2c6Cj4KPiA8ZXJyb3I+Cj4KPiAyMDE3LTA5LTI3IDE2 OjUxOjAyLDc2MC0wNCBFUlJPUgo+IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuYmxsLkRlc3Ryb3lJ bWFnZUNoZWNrQ29tbWFuZF0KPiAoRUUtTWFuYWdlZFRocmVhZEZhY3RvcnktY29tbWFuZENvb3Jk aW5hdG9yLVRocmVhZC02KQo+IFs4OTgxYTBmNC0zYzAyLTRhMmQtYjZiYS1mMjEyMTRiMGViZDRd IFRoZSBmb2xsb3dpbmcgaW1hZ2VzIHdlcmUgbm90Cj4gcmVtb3ZlZDogW2JlMzhkMjRkLTMzYjkt NDNmMC0gOWFiNC0xNWYwMzM0ZTJlYmVdCj4KPiA8L2Vycm9yPgo+Cj4gKgo+Cj4KPiBfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+IERldmVsIG1haWxpbmcg bGlzdAo+IERldmVsQG92aXJ0Lm9yZwo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9s aXN0aW5mby9kZXZlbAoKCgotLS0tLS0tLS0tLS0tLTE3QUVCNTk0OTMzRDQxOEFFOTQ2QUI2MwpD b250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVu Y29kaW5nOiA3Yml0Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRl bnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+CiAg PGJvZHkgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICA8ZGl2IGNsYXNzPSJt b3otY2l0ZS1wcmVmaXgiPmp1c3QgdG8gYWRkIHRoYXQgd2UgaGFkIDIgb3RoZXIKICAgICAgZmFp bHVyZXMgZm9yIGJhc2ljIGRpc2sgb3BlcmF0aW9uIGFmdGVyIHRoZSBvbmUgYmVsb3c6IDxicj4K ICAgICAgPGJyPgo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8v amVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5 MTIvIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVl dWUtdGVzdGVyLzI5MTIvPC9hPjxicj4KPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIg aHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1 ZXVlLXRlc3Rlci8yOTA5LyI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0 ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTA5LzwvYT48YnI+CiAgICAgIDxicj4KICAgICAgYW5k IHRoZSB0ZXN0cyBzdGFydGVkIHBhc3Npbmcgb246IDxicj4KPGEgY2xhc3M9Im1vei10eHQtbGlu ay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0 ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTEzLyI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pv Yi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTEzLzwvYT48YnI+CiAgICAgIDxi cj4KICAgICAgPGJyPgogICAgICBPbiAwOS8yOC8yMDE3IDEyOjA1IFBNLCBEYWZuYSBSb24gd3Jv dGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIgogICAgICBjaXRl PSJtaWQ6Y2E0ZDdkNTAtYzQ1MS00YTUwLTcxMDAtMTUyMzZkMDViZWZmQHJlZGhhdC5jb20iPgog ICAgICA8bWV0YSBodHRwLWVxdWl2PSJjb250ZW50LXR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsg Y2hhcnNldD11dGYtOCI+CiAgICAgIDxwPkhpLCA8YnI+CiAgICAgIDwvcD4KICAgICAgPHA+V2Ug aGFkIGEgZmFpbHVyZSBpbiBPU1QgZm9yIGJhc2ljIGRpc2sgb3BlcmF0aW9uLiA8YnI+CiAgICAg IDwvcD4KICAgICAgPHA+ZnJvbSB3aGF0IEkgY2FuIHNlZSBpbiB0aGUgbG9nLCB0aGVyZSBpcyBh biBpc3N1ZSBkZWxldGluZyBhCiAgICAgICAgc25hcHNob3QgaW1hZ2UgYW5kIHRoZSBpc3N1ZSBp cyByZWxhdGVkIHRvIHRoZSBsaXZlIHN0b3JhZ2UKICAgICAgICBtaWdyYXRpb24uIDxicj4KICAg ICAgPC9wPgogICAgICA8cD5Ib3dldmVyLCBhIHNlY29uZCBwYWlyIG9mIGV5ZXMgd291bGQgaGVs cCBzaW5jZSBJIGNhbiBzZWUKICAgICAgICBlbmdpbmUgcmVwb3J0aW5nIHRoZSBlcnJvciBvbiBm YWlsZWQgZGVsZXRlIHdoaWxlIHZkc20gd2FzIG5vdAogICAgICAgIGFjdHVhbGx5IHJlcG9ydGlu ZyBhIGZhaWxlZCBkZWxldGUsIGF0IGxlYXN0IG5vdCBhcyBFUlJPUi4gPGJyPgogICAgICA8L3A+ CiAgICAgIDxwPkFsc28gSSBkb24ndCB0aGluayB0aGUgcGF0Y2ggdGhhdCBpcyByZXBvcnRlZCBh cyB0aGUgY2F1c2Ugd2FzCiAgICAgICAgYWN0dWFsbHkgdGhlIGNhdXNlLiA8YnI+CiAgICAgIDwv cD4KICAgICAgPHA+SSBncmVwZWQgdGhlIGltYWdlIGZyb20gdGhlIGxvZzogPGEKICAgICAgICAg IGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiCiAgICAgICAgICBocmVmPSJodHRwOi8vcGFz dGViaW4udGVzdC5yZWRoYXQuY29tLzUyMDE1MCIKICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0i dHJ1ZSI+aHR0cDovL3Bhc3RlYmluLnRlc3QucmVkaGF0LmNvbS81MjAxNTA8L2E+PGJyPgogICAg ICA8L3A+CiAgICAgIDxwcmUgd3JhcD0iIj4KPGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbDsi IGlkPSJkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtYzgyNS02NDA1LTYzMzktZTU4NDQwMGY4 NzkyIj48cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7 bWFyZ2luLWJvdHRvbTowcHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1p bHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQt d2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVj b3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw OyI+TGluayB0byBzdXNwZWN0ZWQgcGF0Y2hlczogPC9zcGFuPjwvcD48L2I+PHByZSB3cmFwPSIi PjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHBzOi8vZ2Vycml0Lm92 aXJ0Lm9yZy8jL2MvODIyOTMvMiIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwczovL2dlcnJp dC5vdmlydC5vcmcvIy9jLzgyMjkzLzI8L2E+PC9wcmU+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5v cm1hbDsiIGlkPSJkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtYzgyNS02NDA1LTYzMzktZTU4 NDQwMGY4NzkyIj4KPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10 b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2Zv bnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVu dDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0 ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpw cmUtd3JhcDsiPkxpbmsgdG8gSm9iOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBo cmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVl dWUtdGVzdGVyLzI5MDYiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2plbmtpbnMub3Zp cnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTA2PC9hPjwvc3Bh bj48L3A+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBw dDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZh bWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9u dC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1k ZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdy YXA7Ij5MaW5rIHRvIGFsbCBsb2dzOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBo cmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVl dWUtdGVzdGVyLzI5MDYvYXJ0aWZhY3QvIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9q ZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjkw Ni9hcnRpZmFjdC88L2E+PC9zcGFuPjwvcD4KPHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdo dDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1j b2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12 YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGlu ZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPihSZWxldmFudCkgZXJyb3Igc25pcHBldCBmcm9tIHRo ZSBsb2c6IDwvc3Bhbj48L3A+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21h cmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox MXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFu c3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5v cm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1z cGFjZTpwcmUtd3JhcDsiPiZsdDtlcnJvciZndDs8L3NwYW4+PC9wPgoyMDE3LTA5LTI3IDE2OjUx OjAyLDc2MC0wNCBFUlJPUiBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmJsbC5EZXN0cm95SW1hZ2VD aGVja0NvbW1hbmRdIChFRS1NYW5hZ2VkVGhyZWFkRmFjdG9yeS1jb21tYW5kQ29vcmRpbmF0b3It VGhyZWFkLTYpIFs4OTgxYTBmNC0zYzAyLTRhMmQtYjZiYS1mMjEyMTRiMGViZDRdIFRoZSBmb2xs b3dpbmcgaW1hZ2VzIHdlcmUgbm90IHJlbW92ZWQ6IFtiZTM4ZDI0ZC0zM2I5LTQzZjAtCjlhYjQt MTVmMDMzNGUyZWJlXQoKCjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJn aW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFw dDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3Jt YWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3Bh Y2U6cHJlLXdyYXA7Ij4mbHQ7L2Vycm9yJmd0Ozwvc3Bhbj48L3A+Cgo8L2I+PC9wcmU+CiAgICAg IDxicj4KICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVs ZHNldD4KICAgICAgPGJyPgogICAgICA8cHJlIHdyYXA9IiI+X19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KRGV2ZWwgbWFpbGluZyBsaXN0CjxhIGNsYXNzPSJt b3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Im1haWx0bzpEZXZlbEBvdmlydC5vcmciPkRl dmVsQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0i aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RldmVsIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vZGV2ZWw8L2E+PC9wcmU+CiAgICA8L2Jsb2Nr cXVvdGU+CiAgICA8cD48YnI+CiAgICA8L3A+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0t LS0tLTE3QUVCNTk0OTMzRDQxOEFFOTQ2QUI2My0tCg== --===============3402953656035597619==-- From dron at redhat.com Fri Sep 29 12:46:55 2017 Content-Type: multipart/mixed; boundary="===============5891281288129424629==" MIME-Version: 1.0 From: Dafna Ron To: devel at ovirt.org Subject: Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 28-09-2017 ] [ 004_basic_sanity.disk_operations ] Date: Fri, 29 Sep 2017 13:46:21 +0100 Message-ID: In-Reply-To: ca4d7d50-c451-4a50-7100-15236d05beff@redhat.com --===============5891281288129424629== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------969AC12EBD7819EE5C37AA1C Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit Please note that 004_basic_sanity.disk_operations is still failing: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2941/ thanks, Dafna On 09/28/2017 12:05 PM, Dafna Ron wrote: > > Hi, > > We had a failure in OST for basic disk operation. > > from what I can see in the log, there is an issue deleting a snapshot > image and the issue is related to the live storage migration. > > However, a second pair of eyes would help since I can see engine > reporting the error on failed delete while vdsm was not actually > reporting a failed delete, at least not as ERROR. > > Also I don't think the patch that is reported as the cause was > actually the cause. > > I greped the image from the log: http://pastebin.test.redhat.com/520150 > > * > > Link to suspected patches: > > * > https://gerrit.ovirt.org/#/c/82293/2 > * > > Link to Job: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906 > > Link to all logs: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artifa= ct/ > > (Relevant) error snippet from the log: > > > > 2017-09-27 16:51:02,760-04 ERROR > [org.ovirt.engine.core.bll.DestroyImageCheckCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-6) > [8981a0f4-3c02-4a2d-b6ba-f21214b0ebd4] The following images were not > removed: [be38d24d-33b9-43f0- 9ab4-15f0334e2ebe] > > > > * > > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra --------------969AC12EBD7819EE5C37AA1C Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 7bit

Please note that 004_basic_sanity.disk_operations is still failing:

http://jenkins.ovirt.org/job/ovirt-mas= ter_change-queue-tester/2941/

thanks,
Dafna

On 09/28/2017 12:05 PM, Dafna Ron wrote:

Hi,

We had a failure in OST for basic disk operation.

from what I can see in the log, there is an issue deleting a snapshot image and the issue is related to the live storage migration.

However, a second pair of eyes would help since I can see engine reporting the error on failed delete while vdsm was not actually reporting a failed delete, at least not as ERROR.

Also I don't think the patch that is reported as the cause was actually the cause.

I greped the image from the log: http://pastebin.test.redhat.com/520150
Link to suspected patches: https://gerrit.ovirt.org/#/c/82293/2

Link to Job=
: http://jenki=
ns.ovirt.org/job/ovirt-master_change-queue-tester/2906
Link to all=
 logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2906/artif=
act/
(Relevant) =
error snippet from the log: 
<error>
2017-09-27 16:51:02,760-04 ERROR [org.ovirt.engine.core.bll.DestroyImageChe=
ckCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-6) [8981a0f4-=
3c02-4a2d-b6ba-f21214b0ebd4] The following images were not removed: [be38d2=
4d-33b9-43f0-
9ab4-15f0334e2ebe]


</error&=
gt;
_______________________________________________
Infra mailing list
Inf=
ra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra

--------------969AC12EBD7819EE5C37AA1C-- --===============5891281288129424629== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS05NjlBQzEyRUJENzgxOUVFNUMzN0FBMUMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKClBsZWFzZSBub3RlIHRo YXQgMDA0X2Jhc2ljX3Nhbml0eS5kaXNrX29wZXJhdGlvbnMgaXMgc3RpbGwgZmFpbGluZzoKCmh0 dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0 ZXIvMjk0MS8KCnRoYW5rcywKRGFmbmEKCgpPbiAwOS8yOC8yMDE3IDEyOjA1IFBNLCBEYWZuYSBS b24gd3JvdGU6Cj4KPiBIaSwKPgo+IFdlIGhhZCBhIGZhaWx1cmUgaW4gT1NUIGZvciBiYXNpYyBk aXNrIG9wZXJhdGlvbi4KPgo+IGZyb20gd2hhdCBJIGNhbiBzZWUgaW4gdGhlIGxvZywgdGhlcmUg aXMgYW4gaXNzdWUgZGVsZXRpbmcgYSBzbmFwc2hvdAo+IGltYWdlIGFuZCB0aGUgaXNzdWUgaXMg cmVsYXRlZCB0byB0aGUgbGl2ZSBzdG9yYWdlIG1pZ3JhdGlvbi4KPgo+IEhvd2V2ZXIsIGEgc2Vj b25kIHBhaXIgb2YgZXllcyB3b3VsZCBoZWxwIHNpbmNlIEkgY2FuIHNlZSBlbmdpbmUKPiByZXBv cnRpbmcgdGhlIGVycm9yIG9uIGZhaWxlZCBkZWxldGUgd2hpbGUgdmRzbSB3YXMgbm90IGFjdHVh bGx5Cj4gcmVwb3J0aW5nIGEgZmFpbGVkIGRlbGV0ZSwgYXQgbGVhc3Qgbm90IGFzIEVSUk9SLgo+ Cj4gQWxzbyBJIGRvbid0IHRoaW5rIHRoZSBwYXRjaCB0aGF0IGlzIHJlcG9ydGVkIGFzIHRoZSBj YXVzZSB3YXMKPiBhY3R1YWxseSB0aGUgY2F1c2UuCj4KPiBJIGdyZXBlZCB0aGUgaW1hZ2UgZnJv bSB0aGUgbG9nOiBodHRwOi8vcGFzdGViaW4udGVzdC5yZWRoYXQuY29tLzUyMDE1MAo+Cj4gKgo+ Cj4gTGluayB0byBzdXNwZWN0ZWQgcGF0Y2hlczoKPgo+ICoKPiBodHRwczovL2dlcnJpdC5vdmly dC5vcmcvIy9jLzgyMjkzLzIKPiAqCj4KPiBMaW5rIHRvIEpvYjoKPiBodHRwOi8vamVua2lucy5v dmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MDYKPgo+IExp bmsgdG8gYWxsIGxvZ3M6Cj4gaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0 ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTA2L2FydGlmYWN0Lwo+Cj4gKFJlbGV2YW50KSBlcnJv ciBzbmlwcGV0IGZyb20gdGhlIGxvZzoKPgo+IDxlcnJvcj4KPgo+IDIwMTctMDktMjcgMTY6NTE6 MDIsNzYwLTA0IEVSUk9SCj4gW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuRGVzdHJveUltYWdl Q2hlY2tDb21tYW5kXQo+IChFRS1NYW5hZ2VkVGhyZWFkRmFjdG9yeS1jb21tYW5kQ29vcmRpbmF0 b3ItVGhyZWFkLTYpCj4gWzg5ODFhMGY0LTNjMDItNGEyZC1iNmJhLWYyMTIxNGIwZWJkNF0gVGhl IGZvbGxvd2luZyBpbWFnZXMgd2VyZSBub3QKPiByZW1vdmVkOiBbYmUzOGQyNGQtMzNiOS00M2Yw LSA5YWI0LTE1ZjAzMzRlMmViZV0KPgo+IDwvZXJyb3I+Cj4KPiAqCj4KPgo+IF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gSW5mcmEgbWFpbGluZyBsaXN0 Cj4gSW5mcmFAb3ZpcnQub3JnCj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2luZnJhCgoKCi0tLS0tLS0tLS0tLS0tOTY5QUMxMkVCRDc4MTlFRTVDMzdBQTFDCkNvbnRl bnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rp bmc6IDdiaXQKCjxodG1sPgogIDxoZWFkPgogICAgPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1U eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogIDwvaGVhZD4KICA8Ym9k eSB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxkaXYgY2xhc3M9Im1vei1j aXRlLXByZWZpeCI+UGxlYXNlIG5vdGUgdGhhdAogICAgICAwMDRfYmFzaWNfc2FuaXR5LmRpc2tf b3BlcmF0aW9ucyBpcyBzdGlsbCBmYWlsaW5nOiA8YnI+CiAgICAgIDxicj4KPGEgY2xhc3M9Im1v ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9v dmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTQxLyI+aHR0cDovL2plbmtpbnMub3Zp cnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yOTQxLzwvYT48YnI+ CiAgICAgIDxicj4KICAgICAgdGhhbmtzLCA8YnI+CiAgICAgIERhZm5hPGJyPgogICAgICA8YnI+ CiAgICAgIDxicj4KICAgICAgT24gMDkvMjgvMjAxNyAxMjowNSBQTSwgRGFmbmEgUm9uIHdyb3Rl Ojxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKICAgICAgY2l0ZT0i bWlkOmNhNGQ3ZDUwLWM0NTEtNGE1MC03MTAwLTE1MjM2ZDA1YmVmZkByZWRoYXQuY29tIj4KICAg ICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNo YXJzZXQ9dXRmLTgiPgogICAgICA8cD5IaSwgPGJyPgogICAgICA8L3A+CiAgICAgIDxwPldlIGhh ZCBhIGZhaWx1cmUgaW4gT1NUIGZvciBiYXNpYyBkaXNrIG9wZXJhdGlvbi4gPGJyPgogICAgICA8 L3A+CiAgICAgIDxwPmZyb20gd2hhdCBJIGNhbiBzZWUgaW4gdGhlIGxvZywgdGhlcmUgaXMgYW4g aXNzdWUgZGVsZXRpbmcgYQogICAgICAgIHNuYXBzaG90IGltYWdlIGFuZCB0aGUgaXNzdWUgaXMg cmVsYXRlZCB0byB0aGUgbGl2ZSBzdG9yYWdlCiAgICAgICAgbWlncmF0aW9uLiA8YnI+CiAgICAg IDwvcD4KICAgICAgPHA+SG93ZXZlciwgYSBzZWNvbmQgcGFpciBvZiBleWVzIHdvdWxkIGhlbHAg c2luY2UgSSBjYW4gc2VlCiAgICAgICAgZW5naW5lIHJlcG9ydGluZyB0aGUgZXJyb3Igb24gZmFp bGVkIGRlbGV0ZSB3aGlsZSB2ZHNtIHdhcyBub3QKICAgICAgICBhY3R1YWxseSByZXBvcnRpbmcg YSBmYWlsZWQgZGVsZXRlLCBhdCBsZWFzdCBub3QgYXMgRVJST1IuIDxicj4KICAgICAgPC9wPgog ICAgICA8cD5BbHNvIEkgZG9uJ3QgdGhpbmsgdGhlIHBhdGNoIHRoYXQgaXMgcmVwb3J0ZWQgYXMg dGhlIGNhdXNlIHdhcwogICAgICAgIGFjdHVhbGx5IHRoZSBjYXVzZS4gPGJyPgogICAgICA8L3A+ CiAgICAgIDxwPkkgZ3JlcGVkIHRoZSBpbWFnZSBmcm9tIHRoZSBsb2c6IDxhCiAgICAgICAgICBj bGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IgogICAgICAgICAgaHJlZj0iaHR0cDovL3Bhc3Rl YmluLnRlc3QucmVkaGF0LmNvbS81MjAxNTAiCiAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRy dWUiPmh0dHA6Ly9wYXN0ZWJpbi50ZXN0LnJlZGhhdC5jb20vNTIwMTUwPC9hPjxicj4KICAgICAg PC9wPgogICAgICA8cHJlIHdyYXA9IiI+CjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWw7IiBp ZD0iZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWM4MjUtNjQwNS02MzM5LWU1ODQ0MDBmODc5 MiI+PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21h cmdpbi1ib3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5 OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdl aWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29y YXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsi PkxpbmsgdG8gc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48L3A+PC9iPjxwcmUgd3JhcD0iIj48 YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2dlcnJpdC5vdmly dC5vcmcvIy9jLzgyMjkzLzIiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cHM6Ly9nZXJyaXQu b3ZpcnQub3JnLyMvYy84MjI5My8yPC9hPjwvcHJlPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3Jt YWw7IiBpZD0iZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLWM4MjUtNjQwNS02MzM5LWU1ODQ0 MDBmODc5MiI+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9w OjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250 LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7 Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4 dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJl LXdyYXA7Ij5MaW5rIHRvIEpvYjogPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJl Zj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVl LXRlc3Rlci8yOTA2IiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0 Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjkwNjwvYT48L3NwYW4+ PC9wPgo8cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7 bWFyZ2luLWJvdHRvbTowcHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1p bHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQt d2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVj b3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFw OyI+TGluayB0byBhbGwgbG9nczogPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJl Zj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVl LXRlc3Rlci8yOTA2L2FydGlmYWN0LyIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVu a2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI5MDYv YXJ0aWZhY3QvPC9hPjwvc3Bhbj48L3A+CjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6 MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29s b3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFy aWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7 d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij4oUmVsZXZhbnQpIGVycm9yIHNuaXBwZXQgZnJvbSB0aGUg bG9nOiA8L3NwYW4+PC9wPjxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJn aW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFw dDtmb250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNw YXJlbnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3Jt YWw7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3Bh Y2U6cHJlLXdyYXA7Ij4mbHQ7ZXJyb3ImZ3Q7PC9zcGFuPjwvcD4KMjAxNy0wOS0yNyAxNjo1MTow Miw3NjAtMDQgRVJST1IgW29yZy5vdmlydC5lbmdpbmUuY29yZS5ibGwuRGVzdHJveUltYWdlQ2hl Y2tDb21tYW5kXSAoRUUtTWFuYWdlZFRocmVhZEZhY3RvcnktY29tbWFuZENvb3JkaW5hdG9yLVRo cmVhZC02KSBbODk4MWEwZjQtM2MwMi00YTJkLWI2YmEtZjIxMjE0YjBlYmQ0XSBUaGUgZm9sbG93 aW5nIGltYWdlcyB3ZXJlIG5vdCByZW1vdmVkOiBbYmUzOGQyNGQtMzNiOS00M2YwLQo5YWI0LTE1 ZjAzMzRlMmViZV0KCgo8cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2lu LXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQ7Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7 Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFy ZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFs O3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNl OnByZS13cmFwOyI+Jmx0Oy9lcnJvciZndDs8L3NwYW4+PC9wPgoKPC9iPjwvcHJlPgogICAgICA8 YnI+CiAgICAgIDxmaWVsZHNldCBjbGFzcz0ibWltZUF0dGFjaG1lbnRIZWFkZXIiPjwvZmllbGRz ZXQ+CiAgICAgIDxicj4KICAgICAgPHByZSB3cmFwPSIiPl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fCkluZnJhIG1haWxpbmcgbGlzdAo8YSBjbGFzcz0ibW96 LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86SW5mcmFAb3ZpcnQub3JnIj5JbmZy YUBvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby9pbmZyYSI+aHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2luZnJhPC9hPgo8L3ByZT4KICAgIDwvYmxvY2tx dW90ZT4KICAgIDxwPjxicj4KICAgIDwvcD4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0t LS0tOTY5QUMxMkVCRDc4MTlFRTVDMzdBQTFDLS0K --===============5891281288129424629==--