<div dir="ltr"><div><div><div>Hi, <br><br></div>We have a failure in OST on test 006_migrations.migrate_vm. <br><br></div>From what I can see, the migration succeeded but because there was a KeyError: &#39;cpuUsage&#39; engine assume the v is down and tries to retry migration. <br></div><div>the re-try fails with vm already exists. <br><br><br></div><div><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link and headline of suspected patches: </span></p></b>Failed change: <br><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><a href="https://gerrit.ovirt.org/#/c/88432/2" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>88432/2</a> - </b><br><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446">engine :  Events coming too soon on refresh caps<br><br></b></div><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446">CQ reported this as root ca</b><span style="font-weight:normal">use but I don&#39;t think its related as well: </span><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><br><a href="https://gerrit.ovirt.org/#/c/88404/">https://gerrit.ovirt.org/#/c/88404/</a> - </b><br><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446">core: Auto SD selection on template version update<br><br></b><div><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to Job:</span></p><a href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1021/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1021/</a><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Link to all logs:</span></p><a href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1021/artifact/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/1021/artifact/</a><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(Relevant) error snippet from the log: </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;error&gt;</span></p><br></b></div><div><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446">migration starts and finishs<b> </b></b><span style="font-weight:normal">but fails on vm state on eyError: &#39;cpuUsage&#39; </span><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><br><br><b>2018-03-05 05:21:33,269-0500 INFO  (jsonrpc/3) [api.virt] START migrate(params={u&#39;incomingLimit&#39;: 2, u&#39;tunneled&#39;: u&#39;false&#39;, u&#39;dstqemu&#39;: u&#39;192.0.3.2&#39;, u&#39;autoConverge&#39;: u&#39;false&#39;, u&#39;src&#39;: u&#39;lago-basic-suite-4-2-host-0&#39;, u&#39;enableGuestEvents&#39;:<br> False, u&#39;dst&#39;: u&#39;lago-basic-suite-4-2-host-1:54321&#39;, u&#39;vmId&#39;: u&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;, u&#39;abortOnError&#39;: u&#39;true&#39;, u&#39;outgoingLimit&#39;: 2, u&#39;compressed&#39;: u&#39;false&#39;, u&#39;method&#39;: u&#39;online&#39;}) from=::ffff:192.168.200.2,42452, flow_i<br>d=123f5fd3-81a4-4ba9-9034-525674696629, vmId=a80596a2-f57b-4878-adc3-772363e42783 (api:46)<br>2018-03-05 05:21:33,271-0500 INFO  (jsonrpc/3) [api.virt] FINISH migrate return={&#39;status&#39;: {&#39;message&#39;: &#39;Migration in progress&#39;, &#39;code&#39;: 0}, &#39;progress&#39;: 0} from=::ffff:192.168.200.2,42452, flow_id=123f5fd3-81a4-4ba9-9034-525674696629, vmId<br>=a80596a2-f57b-4878-adc3-772363e42783 (api:52)<br>2018-03-05 05:21:33,271-0500 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.00 seconds (__init__:573)<br>2018-03-05 05:21:33,473-0500 INFO  (monitor/bd2f874) [IOProcessClient] Closing client ioprocess-0 (__init__:583)<br>2018-03-05 05:21:34,936-0500 INFO  (migsrc/a80596a2) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) Creation of destination VM took: 1 seconds (migration:473)<br>2018-03-05 05:21:34,936-0500 INFO  (migsrc/a80596a2) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) starting migration to qemu+tls://lago-basic-suite-4-2-host-1/system with miguri tcp://<a href="http://192.0.3.2">192.0.3.2</a> (migration:502)<br>2018-03-05 05:21:36,356-0500 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.200.2,42452 (api:46)<br>2018-03-05 05:21:36,359-0500 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={&#39;status&#39;: {&#39;message&#39;: &#39;Done&#39;, &#39;code&#39;: 0}, &#39;statsList&#39;: (suppressed)} from=::ffff:192.168.200.2,42452 (api:52)<br>2018-03-05 05:21:36,361-0500 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>2018-03-05 05:21:38,861-0500 INFO  (libvirt/events) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) CPU stopped: onSuspend (vm:6063)<br>2018-03-05 05:21:40,109-0500 WARN  (periodic/0) [virt.periodic.VmDispatcher] could not run &lt;class &#39;vdsm.virt.periodic.DriveWatermarkMonitor&#39;&gt; on [&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;] (periodic:323)<br>2018-03-05 05:21:40,109-0500 INFO  (migsrc/a80596a2) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) migration took 6 seconds to complete (migration:514)<br>2018-03-05 05:21:40,110-0500 INFO  (migsrc/a80596a2) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) Changed state to Down: Migration succeeded (code=4) (vm:1677)<br>2018-03-05 05:21:40,116-0500 INFO  (migsrc/a80596a2) [virt.vm] (vmId=&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;) Stopping connection (guestagent:438)<br>2018-03-05 05:21:41,118-0500 WARN  (periodic/3) [virt.periodic.VmDispatcher] could not run &lt;class &#39;vdsm.virt.periodic.DriveWatermarkMonitor&#39;&gt; on [&#39;a80596a2-f57b-4878-adc3-772363e42783&#39;] (periodic:323)<br>2018-03-05 05:21:42,063-0500 WARN  (periodic/2) [virt.vmstats] Missing stat: &#39;balloon.current&#39; for vm a80596a2-f57b-4878-adc3-772363e42783 (vmstats:552)<br>2018-03-05 05:21:42,064-0500 ERROR (periodic/2) [virt.vmstats] VM metrics collection failed (vmstats:264)<br>Traceback (most recent call last):<br>  File &quot;/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py&quot;, line 197, in send_metrics<br>    data[prefix + &#39;.cpu.usage&#39;] = stat[&#39;cpuUsage&#39;]<br>KeyError: &#39;cpuUsage&#39;<br></b><br></b></div><div><b style="font-weight:normal" id="gmail-docs-internal-guid-e8ba4241-f61d-a080-3b89-946a134e6446"><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">&lt;/error&gt;</span></p></b><br class="gmail-Apple-interchange-newline"></div></div>