[ovirt-devel] Migration failed

Tue Dec 19 08:47:32 UTC 2017

On Tue, Dec 19, 2017 at 9:22 AM, Michal Skrivanek <
michal.skrivanek at redhat.com> wrote:

>
> On 19 Dec 2017, at 07:53, Eyal Edri <eedri at redhat.com> wrote:
>
>
>
> On Dec 19, 2017 00:20, "Michal Skrivanek" <michal.skrivanek at redhat.com>
> wrote:
>
>
> > On 18 Dec 2017, at 13:21, Milan Zamazal <mzamazal at redhat.com> wrote:
> >
> > Yedidyah Bar David <didi at redhat.com> writes:
> >
> >> On Mon, Dec 18, 2017 at 10:17 AM, Code Review <gerrit at ovirt.org> wrote:
> >>> Jenkins CI posted comments on this change.
> >>>
> >>
> >>> View Change
> >>>
> >>> Patch set 3:Continuous-Integration -1
> >>>
> >>> Build Failed
> >>>
> >>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
> -patch-el7-x86_64/2882/
> >>> : FAILURE
> >>
> >> Console output of above job says:
> >>
> >> 08:13:34   # migrate_vm:
> >> 08:16:37     * Collect artifacts:
> >> 08:16:40     * Collect artifacts: Success (in 0:00:03)
> >> 08:16:40   # migrate_vm: Success (in 0:03:06)
> >> 08:16:40   # Results located at
> >> /dev/shm/ost/deployment-basic-suite-master/default/006_migra
> tions.py.junit.xml
> >> 08:16:40 @ Run test: 006_migrations.py: Success (in 0:03:50)
> >> 08:16:40 Error occured, aborting
> >>
> >> The file 006_migrations.py.junit.xml [1] says:
> >>
> >> <failure type="exceptions.AssertionError" message="False != True after
> >> 180 seconds">
> >
> > Reading the logs, I can see the VM migrates normally and seems to be
> > reported to Engine correctly.  When Engine receives end-of-migration
> > event, it sends Destroy to the source (which is correct), calls dumpxmls
> > on the destination in the meantime (looks fine to me) and then calls
>
> looks like a race between getallvmstats reporting VM as Down (statusTime:
> 4296271980) being processed, while there is a Down/MigrationSucceeded event
> arriving (with notify_time 4296273170) at about the same time
> Unfortunately the vdsm.log is not in DEBUG level so there’s very little
> information as to why and what exactly did it send out.
> @infra - can you enable debug log level for vdsm by default?
>
>
> How do you enable debug mode for vdsm ?
>
>
> Can be permanently set in /etc/vdsm/logger.conf
> I suppose enabling for all loggers would be useful
>
> see https://access.redhat.com/articles/2919931
>

Since this file probably is created only after host is added, I assume it
can be set only after add host test, and not in deployment scripts.
So might be needed to added as a test or step after the 'verify add host'
step.

I'll open a ticket on it and we'll see who can assist with it.

>
> Thanks,
> michal
>
>
>
> It does look like a race to me - does it reproduce?
>
> > Destroy on the destination, which is weird and I don't understand why
> > the Destroy is invoked.
> >
> > Arik, would you like to take a look?  Maybe I overlooked something or
> > maybe there's a bug.  The logs are at
> > http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
> -patch-el7-x86_64/2882/artifact/exported-artifacts/basic-
> suite-master__logs/test_logs/basic-suite-master/post-006_migrations.py/
> > and the interesting things happen around 2017-12-18 03:13:43,758-05.
> >
> >> Can someone please have a look? Thanks.
> >>
> >> As a side note, if indeed this is the cause for the failure for the
> >> job, it's misleading to say "migrate_vm: Success".
> >>
> >> [1]
> >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
> -patch-el7-x86_64/2882/artifact/exported-artifacts/basic-
> suite-master__logs/006_migrations.py.junit.xml
> >>
> >>>
> >>> To view, visit change 85177. To unsubscribe, visit settings.
> >>>
> >>> Gerrit-Project: ovirt-system-tests
> >>> Gerrit-Branch: master
> >>> Gerrit-MessageType: comment
> >>> Gerrit-Change-Id: I7eb386744a2a2faf0acd734e0ba44be22dd590b5
> >>> Gerrit-Change-Number: 85177
> >>> Gerrit-PatchSet: 3
> >>> Gerrit-Owner: Yedidyah Bar David <didi at redhat.com>
> >>> Gerrit-Reviewer: Dafna Ron <dron at redhat.com>
> >>> Gerrit-Reviewer: Eyal Edri <eedri at redhat.com>
> >>> Gerrit-Reviewer: Jenkins CI
> >>> Gerrit-Reviewer: Sandro Bonazzola <sbonazzo at redhat.com>
> >>> Gerrit-Reviewer: Yedidyah Bar David <didi at redhat.com>
> >>> Gerrit-Comment-Date: Mon, 18 Dec 2017 08:17:11 +0000
> >>> Gerrit-HasComments: No
> > _______________________________________________
> > Devel mailing list
> > Devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> >
> >
>
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
>
>

-- 

Eyal edri

MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20171219/659493db/attachment.html>