Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 ] [ 2018-04-04 ] [006_migrations.prepare_migration_attachments_ipv6]

On Wed, Apr 25, 2018 at 10:57 AM, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Tue, Apr 24, 2018 at 3:28 PM, Dan Kenigsberg <danken(a)redhat.com&gt; > wrote: > >> On Tue, Apr 24, 2018 at 4:17 PM, Ravi Shankar Nori <rnori(a)redhat.com&gt; >> wrote: >> > >> > >> > On Tue, Apr 24, 2018 at 7:00 AM, Dan Kenigsberg <danken(a)redhat.com&gt; >> wrote: >> >> >> >> Ravi's patch is in, but a similar problem remains, and the test cannot >> >> be put back into its place. >> >> >> >> It seems that while Vdsm was taken down, a couple of getCapsAsync >> >> requests queued up. At one point, the host resumed its connection, >> >> before the requests have been cleared of the queue. After the host is >> >> up, the following tests resume, and at a pseudorandom point in time, >> >> an old getCapsAsync request times out and kills our connection. >> >> >> >> I believe that as long as ANY request is on flight, the monitoring >> >> lock should not be released, and the host should not be declared as >> >> up. >> >> >> >> >> > >> > >> > Hi Dan, >> > >> > Can I have the link to the job on jenkins so I can look at the logs >> >> We disabled a network test that started failing after getCapsAsync was >> merged. >> Please own its re-introduction to OST: >> https://gerrit.ovirt.org/#/c/90264/ >> >> Its most recent failure >> http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/346/ >> has been discussed by Alona and Piotr over IRC. >> > > ​So https://bugzilla.redhat.com/1571768 was created to cover this issue​ > discovered during Alona's and Piotr's conversation. But after further > discussion we have found out that this issue is not related to non-blocking > thread changes in engine 4.2 and this behavior exists from beginning of > vdsm-jsonrpc-java. Ravi will continue verify the fix for BZ1571768 along > with other locking changes he already posted to see if they will help > network OST to succeed. > > But the fix for BZ1571768 is too dangerous for 4.2.3, let's try to fix > that on master and let's see if it doesn't introduce any regressions. If > not, then we can backport to 4.2.4. > > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o. > Posted a vdsm-jsonrpc-java patch [1] for BZ 1571768 [2] which fixes the OST issue with enabling 006_migrations.prepare_migration_attachments_ipv6. I ran OST with the vdsm-jsonrpc-java patch [1] and the patch to add back 006_migrations.prepare_migration_attachments_ipv6 [3] and the jobs succeeded thrice [4][5][6] [1] https://gerrit.ovirt.org/#/c/90646/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1571768 [3] https://gerrit.ovirt.org/#/c/90264/ [4] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2643/ [5] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2644/ [6] http://jenkins.ovirt.org/job/ovirt-system-tests_manual/2645/