<div dir="auto">Ravi/Piotr, so what's the connection between non-blocking threads, jsonrpc-java connection closing and failing this network test? Does it mean that non-blocking threads change just revealed the jsonrpc-java issue which we haven't noticed before?<div dir="auto">And did the test really works with code prior to non-blocking threads changes and we are missing something else? </div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 25 Apr 2018, 18:21 Ravi Shankar Nori, <<a href="mailto:rnori@redhat.com">rnori@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 25, 2018 at 10:57 AM, Martin Perina <span dir="ltr"><<a href="mailto:mperina@redhat.com" target="_blank" rel="noreferrer">mperina@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><div><div class="m_5246604937024840147gmail-h5"><br><div class="gmail_quote">On Tue, Apr 24, 2018 at 3:28 PM, Dan Kenigsberg <span dir="ltr"><<a href="mailto:danken@redhat.com" target="_blank" rel="noreferrer">danken@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Apr 24, 2018 at 4:17 PM, Ravi Shankar Nori <<a href="mailto:rnori@redhat.com" target="_blank" rel="noreferrer">rnori@redhat.com</a>> wrote:<br>
><br>
><br>
> On Tue, Apr 24, 2018 at 7:00 AM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank" rel="noreferrer">danken@redhat.com</a>> wrote:<br>
>><br>
>> Ravi's patch is in, but a similar problem remains, and the test cannot<br>
>> be put back into its place.<br>
>><br>
>> It seems that while Vdsm was taken down, a couple of getCapsAsync<br>
>> requests queued up. At one point, the host resumed its connection,<br>
>> before the requests have been cleared of the queue. After the host is<br>
>> up, the following tests resume, and at a pseudorandom point in time,<br>
>> an old getCapsAsync request times out and kills our connection.<br>
>><br>
>> I believe that as long as ANY request is on flight, the monitoring<br>
>> lock should not be released, and the host should not be declared as<br>
>> up.<br>
>><br>
>><br>
><br>
><br>
> Hi Dan,<br>
><br>
> Can I have the link to the job on jenkins so I can look at the logs<br>
<br>
We disabled a network test that started failing after getCapsAsync was merged.<br>
Please own its re-introduction to OST: <a href="https://gerrit.ovirt.org/#/c/90264/" rel="noreferrer noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/90264/</a><br>
<br>
Its most recent failure<br>
<a href="http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/" rel="noreferrer noreferrer" target="_blank">http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/</a><br>
has been discussed by Alona and Piotr over IRC.<br>
</blockquote></div><br></div></div><div>So <a href="https://bugzilla.redhat.com/1571768" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/1571768</a> was created to cover this issue discovered during Alona's and Piotr's conversation. But after further discussion we have found out that this issue is not related to non-blocking thread changes in engine 4.2 and this behavior exists from beginning of vdsm-jsonrpc-java. Ravi will continue verify the fix for BZ1571768 along with other locking changes he already posted to see if they will help network OST to succeed.<br><br>But the fix for BZ1571768 is too dangerous for 4.2.3, let's try to fix that on master and let's see if it doesn't introduce any regressions. If not, then we can backport to 4.2.4.<br></div><span class="m_5246604937024840147gmail-"><br><br clear="all"><br>-- <br><div class="m_5246604937024840147gmail-m_5735902735463466911gmail_signature"><div dir="ltr"><font size="1">Martin Perina<br>Associate Manager, Software Engineering<br>Red Hat Czech s.r.o.<br></font></div></div>
</span></div></div>
</blockquote></div><br>Posted a vdsm-jsonrpc-java patch [1] for BZ 1571768 [2] which fixes the OST issue with enabling 006_migrations.prepare_migration_attachments_ipv6. <br></div><div class="gmail_extra"><br>I ran OST with the vdsm-jsonrpc-java patch [1] and the patch to add back 006_migrations.prepare_migration_attachments_ipv6 [3] and the jobs succeeded thrice [4][5][6]<br><br>[1] <a href="https://gerrit.ovirt.org/#/c/90646/" target="_blank" rel="noreferrer">https://gerrit.ovirt.org/#/c/90646/</a><br>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1571768" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1571768</a><br>[3] <a href="https://gerrit.ovirt.org/#/c/90264/" target="_blank" rel="noreferrer">https://gerrit.ovirt.org/#/c/90264/</a><br>[4] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2643/" target="_blank" rel="noreferrer">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2643/</a><br>[5] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2644/" target="_blank" rel="noreferrer">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2644/</a><br>[6] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2645/" target="_blank" rel="noreferrer">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2645/</a><br></div></div>
</blockquote></div>