Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 ] [ 2018-04-04 ] [006_migrations.prepare_migration_attachments_ipv6]

Ravi/Piotr, so what's the connection between non-blocking threads, jsonrpc-java connection closing and failing this network test? Does it mean that non-blocking threads change just revealed the jsonrpc-java issue which we haven't noticed before? And did the test really works with code prior to non-blocking threads changes and we are missing something else?

On Wed, 25 Apr 2018, 18:21 Ravi Shankar Nori, <rnori(a)redhat.com&gt; wrote: > > > On Wed, Apr 25, 2018 at 10:57 AM, Martin Perina <mperina(a)redhat.com&gt; > wrote: > >> >> >> On Tue, Apr 24, 2018 at 3:28 PM, Dan Kenigsberg <danken(a)redhat.com&gt; >> wrote: >> >>> On Tue, Apr 24, 2018 at 4:17 PM, Ravi Shankar Nori <rnori(a)redhat.com&gt; >>> wrote: >>> > >>> > >>> > On Tue, Apr 24, 2018 at 7:00 AM, Dan Kenigsberg <danken(a)redhat.com&gt; >>> wrote: >>> >> >>> >> Ravi's patch is in, but a similar problem remains, and the test >>> cannot >>> >> be put back into its place. >>> >> >>> >> It seems that while Vdsm was taken down, a couple of getCapsAsync >>> >> requests queued up. At one point, the host resumed its connection, >>> >> before the requests have been cleared of the queue. After the host is >>> >> up, the following tests resume, and at a pseudorandom point in time, >>> >> an old getCapsAsync request times out and kills our connection. >>> >> >>> >> I believe that as long as ANY request is on flight, the monitoring >>> >> lock should not be released, and the host should not be declared as >>> >> up. >>> >> >>> >> >>> > >>> > >>> > Hi Dan, >>> > >>> > Can I have the link to the job on jenkins so I can look at the logs >>> >>> We disabled a network test that started failing after getCapsAsync was >>> merged. >>> Please own its re-introduction to OST: >>> https://gerrit.ovirt.org/#/c/90264/ >>> >>> Its most recent failure >>> >>> http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/ >>> has been discussed by Alona and Piotr over IRC. >>> >> >> ​So https://bugzilla.redhat.com/1571768 was created to cover this >> issue​ discovered during Alona's and Piotr's conversation. But after >> further discussion we have found out that this issue is not related to >> non-blocking thread changes in engine 4.2 and this behavior exists from >> beginning of vdsm-jsonrpc-java. Ravi will continue verify the fix for >> BZ1571768 along with other locking changes he already posted to see if they >> will help network OST to succeed. >> >> But the fix for BZ1571768 is too dangerous for 4.2.3, let's try to fix >> that on master and let's see if it doesn't introduce any regressions. If >> not, then we can backport to 4.2.4. >> >> >> >> -- >> Martin Perina >> Associate Manager, Software Engineering >> Red Hat Czech s.r.o. >> > > Posted a vdsm-jsonrpc-java patch [1] for BZ 1571768 [2] which fixes the > OST issue with enabling 006_migrations.prepare_migration_attachments_ipv6. > > I ran OST with the vdsm-jsonrpc-java patch [1] and the patch to add back > 006_migrations.prepare_migration_attachments_ipv6 [3] and the jobs > succeeded thrice [4][5][6] > > [1] https://gerrit.ovirt.org/#/c/90646/ > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1571768 > [3] https://gerrit.ovirt.org/#/c/90264/ > [4] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2643/ > [5] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2644/ > [6] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2645/ >