[ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 ] [ 2018-04-04 ] [006_migrations.prepare_migration_attachments_ipv6]

[Piotr Kliczewski]

śr., 25 kwi 2018, 19:07 użytkownik Martin Perina <mperina at redhat.com>
napisał:

> Ravi/Piotr, so what's the connection between non-blocking threads,
> jsonrpc-java connection closing and failing this network test? Does it mean
> that non-blocking threads change just revealed the jsonrpc-java issue which
> we haven't noticed before?
> And did the test really works with code prior to non-blocking threads
> changes and we are missing something else?
>

I think that the test found something not related to non-blocking threads.
This behavior was in the code since the beginning.


>
> On Wed, 25 Apr 2018, 18:21 Ravi Shankar Nori, <rnori at redhat.com> wrote:
>
>>
>>
>> On Wed, Apr 25, 2018 at 10:57 AM, Martin Perina <mperina at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Apr 24, 2018 at 3:28 PM, Dan Kenigsberg <danken at redhat.com>
>>> wrote:
>>>
>>>> On Tue, Apr 24, 2018 at 4:17 PM, Ravi Shankar Nori <rnori at redhat.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Tue, Apr 24, 2018 at 7:00 AM, Dan Kenigsberg <danken at redhat.com>
>>>> wrote:
>>>> >>
>>>> >> Ravi's patch is in, but a similar problem remains, and the test
>>>> cannot
>>>> >> be put back into its place.
>>>> >>
>>>> >> It seems that while Vdsm was taken down, a couple of getCapsAsync
>>>> >> requests queued up. At one point, the host resumed its connection,
>>>> >> before the requests have been cleared of the queue. After the host is
>>>> >> up, the following tests resume, and at a pseudorandom point in time,
>>>> >> an old getCapsAsync request times out and kills our connection.
>>>> >>
>>>> >> I believe that as long as ANY request is on flight, the monitoring
>>>> >> lock should not be released, and the host should not be declared as
>>>> >> up.
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> > Hi Dan,
>>>> >
>>>> > Can I have the link to the job on jenkins so I can look at the logs
>>>>
>>>> We disabled a network test that started failing after getCapsAsync was
>>>> merged.
>>>> Please own its re-introduction to OST:
>>>> https://gerrit.ovirt.org/#/c/90264/
>>>>
>>>> Its most recent failure
>>>>
>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/
>>>> has been discussed by Alona and Piotr over IRC.
>>>>
>>>
>>> ​So https://bugzilla.redhat.com/1571768 was created to cover this
>>> issue​ discovered during Alona's and Piotr's conversation. But after
>>> further discussion we have found out that this issue is not related to
>>> non-blocking thread changes in engine 4.2 and this behavior exists from
>>> beginning of vdsm-jsonrpc-java. Ravi will continue verify the fix for
>>> BZ1571768 along with other locking changes he already posted to see if they
>>> will help network OST to succeed.
>>>
>>> But the fix for BZ1571768 is too dangerous for 4.2.3, let's try to fix
>>> that on master and let's see if it doesn't introduce any regressions. If
>>> not, then we can backport to 4.2.4.
>>>
>>>
>>>
>>> --
>>> Martin Perina
>>> Associate Manager, Software Engineering
>>> Red Hat Czech s.r.o.
>>>
>>
>> Posted a vdsm-jsonrpc-java patch [1] for BZ 1571768 [2] which fixes the
>> OST issue with enabling 006_migrations.prepare_migration_attachments_ipv6.
>>
>> I ran OST with the vdsm-jsonrpc-java patch [1] and the patch to add back
>> 006_migrations.prepare_migration_attachments_ipv6 [3]  and the jobs
>> succeeded thrice [4][5][6]
>>
>> [1] https://gerrit.ovirt.org/#/c/90646/
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1571768
>> [3] https://gerrit.ovirt.org/#/c/90264/
>> [4] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2643/
>> [5] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2644/
>> [6] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2645/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180425/f7fcbf29/attachment-0001.html>