From fromani at redhat.com Wed Jan 11 08:27:03 2017 Content-Type: multipart/mixed; boundary="===============1204629488558966609==" MIME-Version: 1.0 From: Francesco Romani To: devel at ovirt.org Subject: Re: [ovirt-devel] suspend_resume_vm fail on master experimental Date: Wed, 11 Jan 2017 09:26:59 +0100 Message-ID: <7cfce9f0-89a6-5f6f-67e3-797fb52cbd82@redhat.com> In-Reply-To: CACuV_HpRk_wkfvdtJDdZ2sanvvCvb2RY8XSU-bHFCYvHA=0SgA@mail.gmail.com --===============1204629488558966609== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------BC0912C551C51594F7741D03 Content-Type: text/plain; charset=3Dwindows-1252 Content-Transfer-Encoding: 7bit Hi all On 01/11/2017 08:52 AM, Eyal Edri wrote: > Adding Tomas from Virt. > > On Tue, Jan 10, 2017 at 10:54 AM, Piotr Kliczewski > > wrot= e: > > On Tue, Jan 10, 2017 at 9:29 AM, Daniel Belenky > > wrote: > > Hi all, > > > > test-repo_ovirt_experimental_master (link to Jenkins) job failed on > > basic_sanity scenario. > > The job was triggered by https://gerrit.ovirt.org/#/c/69845/ > > > > > From looking at the logs, it seems that the reason is VDSM. > > > > In the VDSM log, i see the following error: > > > > 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) > [vds.dispatcher] SSL > > error receiving from connected ('::1', > > 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119) > Daniel, could you please remind me the jenkins link? I see something suspicious on the Vdsm log. Most notably, Vdsm received SIGTERM. Is this expected and part of the test? > > > > This issue means that the client closed connection while vdsm was > replying. It can happen at any time > when the client is not nice with the connection. As you can see the > client connected locally '::1'. > > > > > Also, when looking at the MOM logs, I see the the following: > > > > 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot > connect to > > VDSM! [Errno 111] Connection refused > > > > Looking at the log at this time vdsm had no open socket. > > Correct, but IIRC we have a race on startup - that's the reason why MOM retries to connect. After the new try, MOM seems to behave correctly: 2017-01-09 16:44:05,672 - mom.RPCServer - INFO - ping() 2017-01-09 16:44:05,673 - mom.RPCServer - INFO - getStatistics() -- = Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------BC0912C551C51594F7741D03 Content-Type: text/html; charset=3Dwindows-1252 Content-Transfer-Encoding: 7bit

Hi all

On 01/11/2017 08:52 AM, Eyal Edri wrote:

Adding Tomas from Virt.

On Tue, Jan 10, 2017 at 10:54 AM, Piotr Kliczewski <piotr.kliczewski(a)gmail.com> wrote:

On Tue, Jan 10, 2017 at 9:29 AM, Daniel Belenky <dbelenky(a)redhat.com= > wrote:
> Hi all,
>
> test-repo_ovirt_experimental_master (link to Jenkins) job failed on
> basic_sanity scenario.
> The job was triggered by https://gerrit.ovirt.o= rg/#/c/69845/
>
> From looking at the logs, it seems that the reason is VDSM.
>
> In the VDSM log, i see the following error:
>
> 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL
> error receiving from <yajsonrpc.betterAsyncore.D= ispatcher connected ('::1',
> 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119)

Daniel, could you please remind me the jenkins link? I see something suspicious on the Vdsm log.
Most notably, Vdsm received SIGTERM. Is this expected and part of the test?

>

This issue means that the client closed connection while vdsm was
replying. It can happen at any time
when the client is not nice with the connection. As you can see the
client connected locally '::1'.

>
> Also, when looking at the MOM logs, I see the the following:
>
> 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot connect to
> VDSM! [Errno 111] Connection refused
>

Looking at the log at this time vdsm had no open socket.

Correct, but IIRC we have a race on startup - that's the reason why MOM retries to connect. After the new try, MOM seems to behave
correctly:

2017-01-09 16:44:05,672 - mom.RPCServer - INFO - ping()
2017-01-09 16:44:05,673 - mom.RPCServer - INFO - getStatistics()

-- =

Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

--------------BC0912C551C51594F7741D03-- --===============1204629488558966609== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1CQzA5MTJDNTUxQzUxNTk0Rjc3NDFEMDMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXdpbmRvd3MtMTI1MgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3Yml0CgpIaSBhbGwK CgpPbiAwMS8xMS8yMDE3IDA4OjUyIEFNLCBFeWFsIEVkcmkgd3JvdGU6Cj4gQWRkaW5nIFRvbWFz IGZyb20gVmlydC4KPgo+IE9uIFR1ZSwgSmFuIDEwLCAyMDE3IGF0IDEwOjU0IEFNLCBQaW90ciBL bGljemV3c2tpCj4gPHBpb3RyLmtsaWN6ZXdza2lAZ21haWwuY29tIDxtYWlsdG86cGlvdHIua2xp Y3pld3NraUBnbWFpbC5jb20+PiB3cm90ZToKPgo+ICAgICBPbiBUdWUsIEphbiAxMCwgMjAxNyBh dCA5OjI5IEFNLCBEYW5pZWwgQmVsZW5reQo+ICAgICA8ZGJlbGVua3lAcmVkaGF0LmNvbSA8bWFp bHRvOmRiZWxlbmt5QHJlZGhhdC5jb20+PiB3cm90ZToKPiAgICAgPiBIaSBhbGwsCj4gICAgID4K PiAgICAgPiB0ZXN0LXJlcG9fb3ZpcnRfZXhwZXJpbWVudGFsX21hc3RlciAobGluayB0byBKZW5r aW5zKSBqb2IgZmFpbGVkIG9uCj4gICAgID4gYmFzaWNfc2FuaXR5IHNjZW5hcmlvLgo+ICAgICA+ IFRoZSBqb2Igd2FzIHRyaWdnZXJlZCBieSBodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzY5 ODQ1Lwo+ICAgICA8aHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy82OTg0NS8+Cj4gICAgID4K PiAgICAgPiBGcm9tIGxvb2tpbmcgYXQgdGhlIGxvZ3MsIGl0IHNlZW1zIHRoYXQgdGhlIHJlYXNv biBpcyBWRFNNLgo+ICAgICA+Cj4gICAgID4gSW4gdGhlIFZEU00gbG9nLCBpIHNlZSB0aGUgZm9s bG93aW5nIGVycm9yOgo+ICAgICA+Cj4gICAgID4gMjAxNy0wMS0wOSAxNjo0Nzo0MSwzMzEgRVJS T1IgKEpzb25ScGMgKFN0b21wUmVhY3RvcikpCj4gICAgIFt2ZHMuZGlzcGF0Y2hlcl0gU1NMCj4g ICAgID4gZXJyb3IgcmVjZWl2aW5nIGZyb20gPHlhanNvbnJwYy5iZXR0ZXJBc3luY29yZS5EaXNw YXRjaGVyCj4gICAgIGNvbm5lY3RlZCAoJzo6MScsCj4gICAgID4gMzQ5NDIsIDAsIDApIGF0IDB4 MzZiOTVmMD46IHVuZXhwZWN0ZWQgZW9mIChiZXR0ZXJBc3luY29yZToxMTkpCj4KCkRhbmllbCwg Y291bGQgeW91IHBsZWFzZSByZW1pbmQgbWUgdGhlIGplbmtpbnMgbGluaz8gSSBzZWUgc29tZXRo aW5nCnN1c3BpY2lvdXMgb24gdGhlIFZkc20gbG9nLgpNb3N0IG5vdGFibHksIFZkc20gcmVjZWl2 ZWQgU0lHVEVSTS4gSXMgdGhpcyBleHBlY3RlZCBhbmQgcGFydCBvZiB0aGUgdGVzdD8KCj4gICAg ID4KPgo+ICAgICBUaGlzIGlzc3VlIG1lYW5zIHRoYXQgdGhlIGNsaWVudCBjbG9zZWQgY29ubmVj dGlvbiB3aGlsZSB2ZHNtIHdhcwo+ICAgICByZXBseWluZy4gSXQgY2FuIGhhcHBlbiBhdCBhbnkg dGltZQo+ICAgICB3aGVuIHRoZSBjbGllbnQgaXMgbm90IG5pY2Ugd2l0aCB0aGUgY29ubmVjdGlv bi4gQXMgeW91IGNhbiBzZWUgdGhlCj4gICAgIGNsaWVudCBjb25uZWN0ZWQgbG9jYWxseSAnOjox Jy4KPgo+ICAgICA+Cj4gICAgID4gQWxzbywgd2hlbiBsb29raW5nIGF0IHRoZSBNT00gbG9ncywg SSBzZWUgdGhlIHRoZSBmb2xsb3dpbmc6Cj4gICAgID4KPiAgICAgPiAyMDE3LTAxLTA5IDE2OjQz OjM5LDUwOCAtIG1vbS52ZHNtSW50ZXJmYWNlIC0gRVJST1IgLSBDYW5ub3QKPiAgICAgY29ubmVj dCB0bwo+ICAgICA+IFZEU00hIFtFcnJubyAxMTFdIENvbm5lY3Rpb24gcmVmdXNlZAo+ICAgICA+ Cj4KPiAgICAgTG9va2luZyBhdCB0aGUgbG9nIGF0IHRoaXMgdGltZSB2ZHNtIGhhZCBubyBvcGVu IHNvY2tldC4KPgo+CgpDb3JyZWN0LCBidXQgSUlSQyB3ZSBoYXZlIGEgcmFjZSBvbiBzdGFydHVw IC0gdGhhdCdzIHRoZSByZWFzb24gd2h5IE1PTQpyZXRyaWVzIHRvIGNvbm5lY3QuIEFmdGVyIHRo ZSBuZXcgdHJ5LCBNT00gc2VlbXMgdG8gYmVoYXZlCmNvcnJlY3RseToKCjIwMTctMDEtMDkgMTY6 NDQ6MDUsNjcyIC0gbW9tLlJQQ1NlcnZlciAtIElORk8gLSBwaW5nKCkKMjAxNy0wMS0wOSAxNjo0 NDowNSw2NzMgLSBtb20uUlBDU2VydmVyIC0gSU5GTyAtIGdldFN0YXRpc3RpY3MoKQoKLS0gCkZy YW5jZXNjbyBSb21hbmkKUmVkIEhhdCBFbmdpbmVlcmluZyBWaXJ0dWFsaXphdGlvbiBSICYgRApJ UkM6IGZyb21hbmkKCgotLS0tLS0tLS0tLS0tLUJDMDkxMkM1NTFDNTE1OTRGNzc0MUQwMwpDb250 ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD13aW5kb3dzLTEyNTIKQ29udGVudC1UcmFuc2Zl ci1FbmNvZGluZzogN2JpdAoKPGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9d2luZG93cy0xMjUyIgogICAgICBodHRwLWVxdWl2PSJDb250ZW50LVR5 cGUiPgogIDwvaGVhZD4KICA8Ym9keSBiZ2NvbG9yPSIjRkZGRkZGIiB0ZXh0PSIjMDAwMDAwIj4K ICAgIDxwPkhpIGFsbDxicj4KICAgIDwvcD4KICAgIDxicj4KICAgIDxkaXYgY2xhc3M9Im1vei1j aXRlLXByZWZpeCI+T24gMDEvMTEvMjAxNyAwODo1MiBBTSwgRXlhbCBFZHJpCiAgICAgIHdyb3Rl Ojxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUKY2l0ZT0ibWlkOkNBQ3VWX0hwUmtfd2tm dmR0SkRkWjJzYW52dkN2YjJSWThYU1UtYkhGQ1l2SEE9MFNnQUBtYWlsLmdtYWlsLmNvbSIKICAg ICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPkFkZGluZyBUb21hcyBmcm9tIFZp cnQuPC9kaXY+CiAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+CiAgICAgICAgPGRp diBjbGFzcz0iZ21haWxfcXVvdGUiPk9uIFR1ZSwgSmFuIDEwLCAyMDE3IGF0IDEwOjU0IEFNLCBQ aW90cgogICAgICAgICAgS2xpY3pld3NraSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIG1vei1kby1u b3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICBocmVmPSJtYWlsdG86cGlvdHIua2xpY3pld3Nr aUBnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5waW90ci5rbGljemV3c2tpQGdtYWlsLmNvbTwv YT4mZ3Q7PC9zcGFuPgogICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgPGJsb2NrcXVvdGUg Y2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAgICAgICAgIC44ZXg7 Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PHNwYW4KICAgICAg ICAgICAgICBjbGFzcz0iIj5PbiBUdWUsIEphbiAxMCwgMjAxNyBhdCA5OjI5IEFNLCBEYW5pZWwg QmVsZW5reQogICAgICAgICAgICAgICZsdDs8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAg ICAgICAgICAgICBocmVmPSJtYWlsdG86ZGJlbGVua3lAcmVkaGF0LmNvbSI+ZGJlbGVua3lAcmVk aGF0LmNvbTwvYT4mZ3Q7CiAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgICAgICZn dDsgSGkgYWxsLDxicj4KICAgICAgICAgICAgICAmZ3Q7PGJyPgogICAgICAgICAgICAgICZndDsg dGVzdC1yZXBvX292aXJ0X2V4cGVyaW1lbnRhbF88d2JyPm1hc3RlciAobGluayB0bwogICAgICAg ICAgICAgIEplbmtpbnMpIGpvYiBmYWlsZWQgb248YnI+CiAgICAgICAgICAgICAgJmd0OyBiYXNp Y19zYW5pdHkgc2NlbmFyaW8uPGJyPgogICAgICAgICAgICAgICZndDsgVGhlIGpvYiB3YXMgdHJp Z2dlcmVkIGJ5IDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgIGhyZWY9 Imh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvNjk4NDUvIgogICAgICAgICAgICAgICAgcmVs PSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMv Yy88d2JyPjY5ODQ1LzwvYT48YnI+CiAgICAgICAgICAgICAgJmd0Ozxicj4KICAgICAgICAgICAg ICAmZ3Q7IEZyb20gbG9va2luZyBhdCB0aGUgbG9ncywgaXQgc2VlbXMgdGhhdCB0aGUgcmVhc29u IGlzCiAgICAgICAgICAgICAgVkRTTS48YnI+CiAgICAgICAgICAgICAgJmd0Ozxicj4KICAgICAg ICAgICAgICAmZ3Q7IEluIHRoZSBWRFNNIGxvZywgaSBzZWUgdGhlIGZvbGxvd2luZyBlcnJvcjo8 YnI+CiAgICAgICAgICAgICAgJmd0Ozxicj4KICAgICAgICAgICAgICAmZ3Q7IDIwMTctMDEtMDkg MTY6NDc6NDEsMzMxIEVSUk9SIChKc29uUnBjCiAgICAgICAgICAgICAgKFN0b21wUmVhY3Rvcikp IFt2ZHMuZGlzcGF0Y2hlcl0gU1NMPGJyPgogICAgICAgICAgICAgICZndDsgZXJyb3IgcmVjZWl2 aW5nIGZyb20gJmx0O3lhanNvbnJwYy5iZXR0ZXJBc3luY29yZS48d2JyPkRpc3BhdGNoZXIKICAg ICAgICAgICAgICBjb25uZWN0ZWQgKCc6OjEnLDxicj4KICAgICAgICAgICAgICAmZ3Q7IDM0OTQy LCAwLCAwKSBhdCAweDM2Yjk1ZjAmZ3Q7OiB1bmV4cGVjdGVkIGVvZgogICAgICAgICAgICAgIChi ZXR0ZXJBc3luY29yZToxMTkpPGJyPgogICAgICAgICAgICA8L3NwYW4+PC9ibG9ja3F1b3RlPgog ICAgICAgIDwvZGl2PgogICAgICA8L2Rpdj4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxicj4KICAg IERhbmllbCwgY291bGQgeW91IHBsZWFzZSByZW1pbmQgbWUgdGhlIGplbmtpbnMgbGluaz8gSSBz ZWUgc29tZXRoaW5nCiAgICBzdXNwaWNpb3VzIG9uIHRoZSBWZHNtIGxvZy48YnI+CiAgICBNb3N0 IG5vdGFibHksIFZkc20gcmVjZWl2ZWQgU0lHVEVSTS4gSXMgdGhpcyBleHBlY3RlZCBhbmQgcGFy dCBvZgogICAgdGhlIHRlc3Q/PGJyPgogICAgPGJyPgogICAgPGJsb2NrcXVvdGUKY2l0ZT0ibWlk OkNBQ3VWX0hwUmtfd2tmdmR0SkRkWjJzYW52dkN2YjJSWThYU1UtYkhGQ1l2SEE9MFNnQUBtYWls LmdtYWlsLmNvbSIKICAgICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4 dHJhIj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAgICAgICAgICA8YmxvY2tx dW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAg LjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48c3Bhbgog ICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgICZndDs8YnI+CiAgICAgICAgICAg ICAgPGJyPgogICAgICAgICAgICA8L3NwYW4+VGhpcyBpc3N1ZSBtZWFucyB0aGF0IHRoZSBjbGll bnQgY2xvc2VkIGNvbm5lY3Rpb24KICAgICAgICAgICAgd2hpbGUgdmRzbSB3YXM8YnI+CiAgICAg ICAgICAgIHJlcGx5aW5nLiBJdCBjYW4gaGFwcGVuIGF0IGFueSB0aW1lPGJyPgogICAgICAgICAg ICB3aGVuIHRoZSBjbGllbnQgaXMgbm90IG5pY2Ugd2l0aCB0aGUgY29ubmVjdGlvbi4gQXMgeW91 IGNhbgogICAgICAgICAgICBzZWUgdGhlPGJyPgogICAgICAgICAgICBjbGllbnQgY29ubmVjdGVk IGxvY2FsbHkgJzo6MScuPGJyPgogICAgICAgICAgICA8c3BhbiBjbGFzcz0iIj48YnI+CiAgICAg ICAgICAgICAgJmd0Ozxicj4KICAgICAgICAgICAgICAmZ3Q7IEFsc28sIHdoZW4gbG9va2luZyBh dCB0aGUgTU9NIGxvZ3MsIEkgc2VlIHRoZSB0aGUKICAgICAgICAgICAgICBmb2xsb3dpbmc6PGJy PgogICAgICAgICAgICAgICZndDs8YnI+CiAgICAgICAgICAgICAgJmd0OyAyMDE3LTAxLTA5IDE2 OjQzOjM5LDUwOCAtIG1vbS52ZHNtSW50ZXJmYWNlIC0gRVJST1IgLQogICAgICAgICAgICAgIENh bm5vdCBjb25uZWN0IHRvPGJyPgogICAgICAgICAgICAgICZndDsgVkRTTSEgW0Vycm5vIDExMV0g Q29ubmVjdGlvbiByZWZ1c2VkPGJyPgogICAgICAgICAgICAgICZndDs8YnI+CiAgICAgICAgICAg ICAgPGJyPgogICAgICAgICAgICA8L3NwYW4+TG9va2luZyBhdCB0aGUgbG9nIGF0IHRoaXMgdGlt ZSB2ZHNtIGhhZCBubyBvcGVuCiAgICAgICAgICAgIHNvY2tldC48L2Jsb2NrcXVvdGU+CiAgICAg ICAgPC9kaXY+CiAgICAgIDwvZGl2PgogICAgICA8YnI+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8 YnI+CiAgICBDb3JyZWN0LCBidXQgSUlSQyB3ZSBoYXZlIGEgcmFjZSBvbiBzdGFydHVwIC0gdGhh dCdzIHRoZSByZWFzb24gd2h5CiAgICBNT00gcmV0cmllcyB0byBjb25uZWN0LiBBZnRlciB0aGUg bmV3IHRyeSwgTU9NIHNlZW1zIHRvIGJlaGF2ZTxicj4KICAgIGNvcnJlY3RseTo8YnI+CiAgICA8 YnI+CiAgICAyMDE3LTAxLTA5IDE2OjQ0OjA1LDY3MiAtIG1vbS5SUENTZXJ2ZXIgLSBJTkZPIC0g cGluZygpPGJyPgogICAgMjAxNy0wMS0wOSAxNjo0NDowNSw2NzMgLSBtb20uUlBDU2VydmVyIC0g SU5GTyAtIGdldFN0YXRpc3RpY3MoKTxicj4KICAgIDxicj4KICAgIDxwcmUgY2xhc3M9Im1vei1z aWduYXR1cmUiIGNvbHM9IjcyIj4tLSAKRnJhbmNlc2NvIFJvbWFuaQpSZWQgSGF0IEVuZ2luZWVy aW5nIFZpcnR1YWxpemF0aW9uIFIgJmFtcDsgRApJUkM6IGZyb21hbmk8L3ByZT4KICA8L2JvZHk+ CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0tQkMwOTEyQzU1MUM1MTU5NEY3NzQxRDAzLS0K --===============1204629488558966609==--