[ovirt-devel] Migration failed

[Eyal Edri]

On Dec 19, 2017 00:20, "Michal Skrivanek" <michal.skrivanek at redhat.com>
wrote:


> On 18 Dec 2017, at 13:21, Milan Zamazal <mzamazal at redhat.com> wrote:
>
> Yedidyah Bar David <didi at redhat.com> writes:
>
>> On Mon, Dec 18, 2017 at 10:17 AM, Code Review <gerrit at ovirt.org> wrote:
>>> Jenkins CI posted comments on this change.
>>>
>>
>>> View Change
>>>
>>> Patch set 3:Continuous-Integration -1
>>>
>>> Build Failed
>>>
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_
check-patch-el7-x86_64/2882/
>>> : FAILURE
>>
>> Console output of above job says:
>>
>> 08:13:34   # migrate_vm:
>> 08:16:37     * Collect artifacts:
>> 08:16:40     * Collect artifacts: Success (in 0:00:03)
>> 08:16:40   # migrate_vm: Success (in 0:03:06)
>> 08:16:40   # Results located at
>> /dev/shm/ost/deployment-basic-suite-master/default/006_
migrations.py.junit.xml
>> 08:16:40 @ Run test: 006_migrations.py: Success (in 0:03:50)
>> 08:16:40 Error occured, aborting
>>
>> The file 006_migrations.py.junit.xml [1] says:
>>
>> <failure type="exceptions.AssertionError" message="False != True after
>> 180 seconds">
>
> Reading the logs, I can see the VM migrates normally and seems to be
> reported to Engine correctly.  When Engine receives end-of-migration
> event, it sends Destroy to the source (which is correct), calls dumpxmls
> on the destination in the meantime (looks fine to me) and then calls

looks like a race between getallvmstats reporting VM as Down (statusTime:
4296271980) being processed, while there is a Down/MigrationSucceeded event
arriving (with notify_time 4296273170) at about the same time
Unfortunately the vdsm.log is not in DEBUG level so there’s very little
information as to why and what exactly did it send out.
@infra - can you enable debug log level for vdsm by default?


How do you enable debug mode for vdsm ?


It does look like a race to me - does it reproduce?

> Destroy on the destination, which is weird and I don't understand why
> the Destroy is invoked.
>
> Arik, would you like to take a look?  Maybe I overlooked something or
> maybe there's a bug.  The logs are at
> http://jenkins.ovirt.org/job/ovirt-system-tests_master_
check-patch-el7-x86_64/2882/artifact/exported-artifacts/
basic-suite-master__logs/test_logs/basic-suite-master/post-
006_migrations.py/
> and the interesting things happen around 2017-12-18 03:13:43,758-05.
>
>> Can someone please have a look? Thanks.
>>
>> As a side note, if indeed this is the cause for the failure for the
>> job, it's misleading to say "migrate_vm: Success".
>>
>> [1]
>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_
check-patch-el7-x86_64/2882/artifact/exported-artifacts/
basic-suite-master__logs/006_migrations.py.junit.xml
>>
>>>
>>> To view, visit change 85177. To unsubscribe, visit settings.
>>>
>>> Gerrit-Project: ovirt-system-tests
>>> Gerrit-Branch: master
>>> Gerrit-MessageType: comment
>>> Gerrit-Change-Id: I7eb386744a2a2faf0acd734e0ba44be22dd590b5
>>> Gerrit-Change-Number: 85177
>>> Gerrit-PatchSet: 3
>>> Gerrit-Owner: Yedidyah Bar David <didi at redhat.com>
>>> Gerrit-Reviewer: Dafna Ron <dron at redhat.com>
>>> Gerrit-Reviewer: Eyal Edri <eedri at redhat.com>
>>> Gerrit-Reviewer: Jenkins CI
>>> Gerrit-Reviewer: Sandro Bonazzola <sbonazzo at redhat.com>
>>> Gerrit-Reviewer: Yedidyah Bar David <didi at redhat.com>
>>> Gerrit-Comment-Date: Mon, 18 Dec 2017 08:17:11 +0000
>>> Gerrit-HasComments: No
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
>

_______________________________________________
Infra mailing list
Infra at ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20171219/43df4699/attachment.html>