This is a multi-part message in MIME format. --------------0EBD452C6DE2C2E44A2F831B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Thanks, sorry, I was silly enough to have it missed before. On 01/11/2017 09:32 AM, Daniel Belenky wrote:
On Wed, Jan 11, 2017 at 10:26 AM, Francesco Romani <fromani@redhat.com <mailto:fromani@redhat.com>> wrote:
Hi all
On 01/11/2017 08:52 AM, Eyal Edri wrote:
Adding Tomas from Virt.
On Tue, Jan 10, 2017 at 10:54 AM, Piotr Kliczewski <piotr.kliczewski@gmail.com <mailto:piotr.kliczewski@gmail.com>> wrote:
On Tue, Jan 10, 2017 at 9:29 AM, Daniel Belenky <dbelenky@redhat.com <mailto:dbelenky@redhat.com>> wrote: > Hi all, > > test-repo_ovirt_experimental_master (link to Jenkins) job failed on > basic_sanity scenario. > The job was triggered by https://gerrit.ovirt.org/#/c/69845/ <https://gerrit.ovirt.org/#/c/69845/> > > From looking at the logs, it seems that the reason is VDSM. > > In the VDSM log, i see the following error: > > 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL > error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', > 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119)
Daniel, could you please remind me the jenkins link? I see something suspicious on the Vdsm log. Most notably, Vdsm received SIGTERM. Is this expected and part of the test?
>
This issue means that the client closed connection while vdsm was replying. It can happen at any time when the client is not nice with the connection. As you can see the client connected locally '::1'.
> > Also, when looking at the MOM logs, I see the the following: > > 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot connect to > VDSM! [Errno 111] Connection refused >
Looking at the log at this time vdsm had no open socket.
Correct, but IIRC we have a race on startup - that's the reason why MOM retries to connect. After the new try, MOM seems to behave correctly:
2017-01-09 16:44:05,672 - mom.RPCServer - INFO - ping() 2017-01-09 16:44:05,673 - mom.RPCServer - INFO - getStatistics()
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani
-- /Daniel Belenky / /RHV DevOps / /Red Hat Israel /
-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------0EBD452C6DE2C2E44A2F831B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Thanks, sorry, I was silly enough to have it missed before.<br> </p> <br> <div class="moz-cite-prefix">On 01/11/2017 09:32 AM, Daniel Belenky wrote:<br> </div> <blockquote cite="mid:CACDJhnfxXuzOYBU3CcdRrsZSi6+pyZn9Ot8ZJgDnw+mTPDqsUA@mail.gmail.com" type="cite"> <div dir="ltr"><a moz-do-not-send="true" href="http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_master/4648/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/">Link to Jenkins</a></div> <div class="gmail_extra"><br> <div class="gmail_quote">On Wed, Jan 11, 2017 at 10:26 AM, Francesco Romani <span dir="ltr"><<a moz-do-not-send="true" href="mailto:fromani@redhat.com" target="_blank">fromani@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <p>Hi all<br> </p> <span class=""> <br> <div class="m_1420320669945181361moz-cite-prefix">On 01/11/2017 08:52 AM, Eyal Edri wrote:<br> </div> <blockquote type="cite"> <div dir="ltr">Adding Tomas from Virt.</div> <div class="gmail_extra"><br> <div class="gmail_quote">On Tue, Jan 10, 2017 at 10:54 AM, Piotr Kliczewski <span dir="ltr"><<a moz-do-not-send="true" href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Jan 10, 2017 at 9:29 AM, Daniel Belenky <<a moz-do-not-send="true" href="mailto:dbelenky@redhat.com" target="_blank">dbelenky@redhat.com</a>> wrote:<br> > Hi all,<br> ><br> > test-repo_ovirt_experimental_m<wbr>aster (link to Jenkins) job failed on<br> > basic_sanity scenario.<br> > The job was triggered by <a moz-do-not-send="true" href="https://gerrit.ovirt.org/#/c/69845/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>9845/</a><br> ><br> > From looking at the logs, it seems that the reason is VDSM.<br> ><br> > In the VDSM log, i see the following error:<br> ><br> > 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL<br> > error receiving from <yajsonrpc.betterAsyncore.Disp<wbr>atcher connected ('::1',<br> > 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119)<br> </span></blockquote> </div> </div> </blockquote> <br> </span> Daniel, could you please remind me the jenkins link? I see something suspicious on the Vdsm log.<br> Most notably, Vdsm received SIGTERM. Is this expected and part of the test?<span class=""><br> <br> <blockquote type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span> ><br> <br> </span>This issue means that the client closed connection while vdsm was<br> replying. It can happen at any time<br> when the client is not nice with the connection. As you can see the<br> client connected locally '::1'.<br> <span><br> ><br> > Also, when looking at the MOM logs, I see the the following:<br> ><br> > 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot connect to<br> > VDSM! [Errno 111] Connection refused<br> ><br> <br> </span>Looking at the log at this time vdsm had no open socket.</blockquote> </div> </div> <br> </blockquote> <br> </span> Correct, but IIRC we have a race on startup - that's the reason why MOM retries to connect. After the new try, MOM seems to behave<br> correctly:<br> <br> 2017-01-09 16:44:05,672 - mom.RPCServer - INFO - ping()<br> 2017-01-09 16:44:05,673 - mom.RPCServer - INFO - getStatistics()<span class="HOEnZb"><font color="#888888"><br> <br> <pre class="m_1420320669945181361moz-signature" cols="72">-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani</pre> </font></span></div> </blockquote> </div> <br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail_signature" data-smartmail="gmail_signature"> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <div> <div><span style="background-color:rgb(255,255,255)"><span style="color:rgb(0,0,0)"><i>Daniel Belenky<br> </i></span></span></div> <span style="background-color:rgb(255,255,255)"><span style="color:rgb(0,0,0)"><i>RHV DevOps<br> </i></span></span></div> <span style="background-color:rgb(255,255,255)"><span style="color:rgb(0,0,0)"><i>Red Hat Israel<br> </i></span></span></div> </div> </div> </div> </div> </div> </div> </div> </blockquote> <br> <pre class="moz-signature" cols="72">-- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani</pre> </body> </html> --------------0EBD452C6DE2C2E44A2F831B--