<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 24, 2018 at 10:46 AM, Ravi Shankar Nori <span dir="ltr"><<a href="mailto:rnori@redhat.com" target="_blank">rnori@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Tue, Apr 24, 2018 at 10:29 AM, Dan Kenigsberg <span dir="ltr"><<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Apr 24, 2018 at 5:09 PM, Ravi Shankar Nori <<a href="mailto:rnori@redhat.com" target="_blank">rnori@redhat.com</a>> wrote:<br>
><br>
><br>
> On Tue, Apr 24, 2018 at 9:47 AM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>> wrote:<br>
>><br>
>> On Tue, Apr 24, 2018 at 4:36 PM, Ravi Shankar Nori <<a href="mailto:rnori@redhat.com" target="_blank">rnori@redhat.com</a>><br>
>> wrote:<br>
>> ><br>
>> ><br>
>> > On Tue, Apr 24, 2018 at 9:24 AM, Martin Perina <<a href="mailto:mperina@redhat.com" target="_blank">mperina@redhat.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >><br>
>> >><br>
>> >> On Tue, Apr 24, 2018 at 3:17 PM, Ravi Shankar Nori <<a href="mailto:rnori@redhat.com" target="_blank">rnori@redhat.com</a>><br>
>> >> wrote:<br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>> On Tue, Apr 24, 2018 at 7:00 AM, Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>><br>
>> >>> wrote:<br>
>> >>>><br>
>> >>>> Ravi's patch is in, but a similar problem remains, and the test<br>
>> >>>> cannot<br>
>> >>>> be put back into its place.<br>
>> >>>><br>
>> >>>> It seems that while Vdsm was taken down, a couple of getCapsAsync<br>
>> >>>> requests queued up. At one point, the host resumed its connection,<br>
>> >>>> before the requests have been cleared of the queue. After the host is<br>
>> >>>> up, the following tests resume, and at a pseudorandom point in time,<br>
>> >>>> an old getCapsAsync request times out and kills our connection.<br>
>> >>>><br>
>> >>>> I believe that as long as ANY request is on flight, the monitoring<br>
>> >>>> lock should not be released, and the host should not be declared as<br>
>> >>>> up.<br>
>><br>
>> Would you relate to this analysis ^^^ ?<br>
>><br>
><br>
> The HostMonitoring lock issue has been fixed by<br>
> <a href="https://gerrit.ovirt.org/#/c/90189/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/9<wbr>0189/</a><br>
<br>
</span>Is there still a chance that a host moves to Up while former<br>
getCapsAsync request are still in-flight?<br>
<div class="m_3961909441223816395HOEnZb"><div class="m_3961909441223816395h5"><br></div></div></blockquote><div><br></div></div></div><div>Should not happen. Is there a way to execute/reproduce the failing test on Dev env?<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_3961909441223816395HOEnZb"><div class="m_3961909441223816395h5">
><br>
>><br>
>> >>>><br>
>> >>>><br>
>> >>><br>
>> >>><br>
>> >>> Hi Dan,<br>
>> >>><br>
>> >>> Can I have the link to the job on jenkins so I can look at the logs<br>
>> >><br>
>> >><br>
>> >><br>
>> >> <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt-system-tests_standard-che<wbr>ck-patch/346/</a><br>
>> >><br>
>> ><br>
>> ><br>
>> > From the logs the only VDS lock that is being released twice is<br>
>> > VDS_FENCE<br>
>> > lock. Opened a BZ [1] for it. Will post a fix<br>
>> ><br>
>> > [1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1571300" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1571300</a><br>
>><br>
>> Can this possibly cause a surprise termination of host connection?<br>
><br>
><br>
> Not sure, from the logs VDS_FENCE is the only other VDS lock that is being<br>
> released<br>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div><br></div><div class="gmail_extra">Would be helpful if I can get the exact flow that is failing and also the steps if any needed to reproduce the issue<br></div></div>