Re: [ovirt-devel] Migration failed

On 19 Dec 2017, at 07:53, Eyal Edri <eedri(a)redhat.com&gt; wrote: On Dec 19, 2017 00:20, "Michal Skrivanek" <michal.skrivanek(a)redhat.com&gt; wrote: > On 18 Dec 2017, at 13:21, Milan Zamazal <mzamazal(a)redhat.com&gt; wrote: > > Yedidyah Bar David <didi(a)redhat.com&gt; writes: > >> On Mon, Dec 18, 2017 at 10:17 AM, Code Review <gerrit(a)ovirt.org&gt; wrote: >>> Jenkins CI posted comments on this change. >>> >> >>> View Change >>> >>> Patch set 3:Continuous-Integration -1 >>> >>> Build Failed >>> >>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check -patch-el7-x86_64/2882/ >>> : FAILURE >> >> Console output of above job says: >> >> 08:13:34 # migrate_vm: >> 08:16:37 * Collect artifacts: >> 08:16:40 * Collect artifacts: Success (in 0:00:03) >> 08:16:40 # migrate_vm: Success (in 0:03:06) >> 08:16:40 # Results located at >> /dev/shm/ost/deployment-basic-suite-master/default/006_migra tions.py.junit.xml >> 08:16:40 @ Run test: 006_migrations.py: Success (in 0:03:50) >> 08:16:40 Error occured, aborting >> >> The file 006_migrations.py.junit.xml [1] says: >> >> <failure type="exceptions.AssertionError" message="False != True after >> 180 seconds"> > > Reading the logs, I can see the VM migrates normally and seems to be > reported to Engine correctly. When Engine receives end-of-migration > event, it sends Destroy to the source (which is correct), calls dumpxmls > on the destination in the meantime (looks fine to me) and then calls looks like a race between getallvmstats reporting VM as Down (statusTime: 4296271980) being processed, while there is a Down/MigrationSucceeded event arriving (with notify_time 4296273170) at about the same time Unfortunately the vdsm.log is not in DEBUG level so there’s very little information as to why and what exactly did it send out. @infra - can you enable debug log level for vdsm by default? How do you enable debug mode for vdsm ? Can be permanently set in /etc/vdsm/logger.conf I suppose enabling for all loggers would be useful see https://access.redhat.com/articles/2919931

Thanks, michal It does look like a race to me - does it reproduce? > Destroy on the destination, which is weird and I don't understand why > the Destroy is invoked. > > Arik, would you like to take a look? Maybe I overlooked something or > maybe there's a bug. The logs are at > http://jenkins.ovirt.org/job/ovirt-system-tests_master_check -patch-el7-x86_64/2882/artifact/exported-artifacts/basic- suite-master__logs/test_logs/basic-suite-master/post-006_migrations.py/ > and the interesting things happen around 2017-12-18 03:13:43,758-05. > >> Can someone please have a look? Thanks. >> >> As a side note, if indeed this is the cause for the failure for the >> job, it's misleading to say "migrate_vm: Success". >> >> [1] >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check -patch-el7-x86_64/2882/artifact/exported-artifacts/basic- suite-master__logs/006_migrations.py.junit.xml >> >>> >>> To view, visit change 85177. To unsubscribe, visit settings. >>> >>> Gerrit-Project: ovirt-system-tests >>> Gerrit-Branch: master >>> Gerrit-MessageType: comment >>> Gerrit-Change-Id: I7eb386744a2a2faf0acd734e0ba44be22dd590b5 >>> Gerrit-Change-Number: 85177 >>> Gerrit-PatchSet: 3 >>> Gerrit-Owner: Yedidyah Bar David <didi(a)redhat.com&gt; >>> Gerrit-Reviewer: Dafna Ron <dron(a)redhat.com&gt; >>> Gerrit-Reviewer: Eyal Edri <eedri(a)redhat.com&gt; >>> Gerrit-Reviewer: Jenkins CI >>> Gerrit-Reviewer: Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>> Gerrit-Reviewer: Yedidyah Bar David <didi(a)redhat.com&gt; >>> Gerrit-Comment-Date: Mon, 18 Dec 2017 08:17:11 +0000 >>> Gerrit-HasComments: No > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > > _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra