<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 18 Jan 2018, at 17:36, Arik Hadas <<a href="mailto:ahadas@redhat.com" class="">ahadas@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Jan 17, 2018 at 9:41 PM, Milan Zamazal <span dir="ltr" class=""><<a href="mailto:mzamazal@redhat.com" target="_blank" class="">mzamazal@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span class="gmail-">Dafna Ron <<a href="mailto:dron@redhat.com" class="">dron@redhat.com</a>> writes:<br class="">
<br class="">
> We had a failure in test 006_migrations.migrate_vm<br class="">
</span>> <<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4842/testReport/junit/%28root%29/006_migrations/migrate_vm/" rel="noreferrer" target="_blank" class="">http://jenkins.ovirt.org/job/<wbr class="">ovirt-master_change-queue-<wbr class="">tester/4842/testReport/junit/%<wbr class="">28root%29/006_migrations/<wbr class="">migrate_vm/</a>>.<br class="">
<span class="gmail-">><br class="">
> the migration failed with reason "VMExists"<br class="">
<br class="">
</span>There are two migrations in 006_migrations.migrate_vm. The first one<br class="">
succeeded, but if I'm looking correctly into the logs, Engine didn't<br class="">
send Destroy to the source host after the migration had finished. Then<br class="">
the second migration gets rejected by Vdsm, because Vdsm still keeps the<br class="">
former Vm object instance in Down status.<br class="">
<br class="">
Since the test succeeds most of the time, it looks like some timing<br class="">
issue or border case. Arik, is it a known problem? If not, would you<br class="">
like to look into the logs, whether you can see what's happening?</blockquote><div class=""><br class=""></div><div class="">Your analysis is correct. That's a nice one actually!</div><div class=""><br class=""></div><div class="">The statistics monitoring cycles of both hosts host-0 and host-1 were scheduled in a way that they are executed almost at the same time [1].</div><div class=""><br class=""></div><div class="">Now, at 6:46:34 the VM was migrated from host-1 to host-0.</div><div class="">At 6:46:42 the migration succeeded - we got events from both hosts, but only processed the one from the destination so the VM switched to Up.</div><div class="">The next statistics monitoring cycle was triggered at 6:46:44 - again, the report of that VM from the source host was skipped because we processed the one from the destination.</div><div class="">At 6:46:59, in the next statistics monitoring cycle, it happened again - the report of the VM from the source host was skipped.</div><div class="">The next migration was triggered at 6:47:05 - the engine didn't manage to process any report from the source host, so the VM remained Down there. </div><div class=""><br class=""></div><div class="">The probability of this to happen is extremely low.</div></div></div></div></div></blockquote><div><br class=""></div></div><div>Why wasn't the migration rerun?</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">However, I think we can make a little tweak to the monitoring code to avoid this:</div><div class="">"If we get the VM as Down on an unexpected host (that is, not the host we expect the VM to run on), do not lock the VM"</div><div class="">It should be safe since we don't update anything in this scenario.</div><div class=""> </div><div class="">[1] For instance:</div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">2018-01-15 06:46:44,905-05 ... </span>GetAllVmStatsVDSCommand ... VdsIdVDSCommandParametersBase:{hostId='873a4d36-55fe-4be1-acb7-8de9c9123eb2'})</div></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">2018-01-15 06:46:44,932-05 ... </span>GetAllVmStatsVDSCommand ... VdsIdVDSCommandParametersBase:{hostId='31f09289-ec6c-42ff-a745-e82e8ac8e6b9'})</div></div></div></div></div>
_______________________________________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" class="">Devel@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote></div><br class=""></body></html>