<div dir="ltr">And this is the vdsm log from vmhost04:<br><br>Thread-6320715::DEBUG::2017-03-24 11:38:02,224::migration::188::virt.vm::(_setupVdsConnection) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Initiating connection with destination<br>Thread-6320715::DEBUG::2017-03-24 11:38:02,239::migration::200::virt.vm::(_setupVdsConnection) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Destination server is: vm08.ovirt.internal:54321<br>Thread-6320715::DEBUG::2017-03-24 11:38:02,241::migration::246::virt.vm::(_prepareGuest) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration started<br>Thread-6320715::DEBUG::2017-03-24 11:38:02,253::guestagent::502::virt.vm::(send_lifecycle_event) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::send_lifecycle_event before_migration called<br>Thread-6320715::DEBUG::2017-03-24 11:38:02,319::migration::353::virt.vm::(run) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::migration semaphore acquired after 2 seconds<br>Thread-6320715::INFO::2017-03-24 11:38:02,950::migration::407::virt.vm::(_startUnderlyingMigration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Creation of destination VM took: 0 seconds<br>Thread-6320715::INFO::2017-03-24 11:38:02,951::migration::429::virt.vm::(_startUnderlyingMigration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::starting migration to qemu+tls://vm08.ovirt.internal/system with miguri tcp://<a href="http://192.168.10.107">192.168.10.107</a><br>Thread-6320715::DEBUG::2017-03-24 11:38:02,951::migration::494::virt.vm::(_perform_with_conv_schedule) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::performing migration with conv schedule<br>Thread-6320717::DEBUG::2017-03-24 11:38:02,952::migration::620::virt.vm::(run) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::starting migration monitor thread<br>Thread-6320717::DEBUG::2017-03-24 11:38:02,952::migration::739::virt.vm::(_execute_action_with_params) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Setting downtime to 100<br>Thread-6320717::INFO::2017-03-24 11:38:12,959::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 10 seconds elapsed, 8% of data processed, total data: 16456MB, processed data: 483MB, remaining data: 15157MB, transfer speed 52MBps, zero pages: 209524MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:38:22,963::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 20 seconds elapsed, 13% of data processed, total data: 16456MB, processed data: 1003MB, remaining data: 14469MB, transfer speed 51MBps, zero pages: 252844MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:38:32,966::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 30 seconds elapsed, 16% of data processed, total data: 16456MB, processed data: 1524MB, remaining data: 13839MB, transfer speed 52MBps, zero pages: 281344MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:38:42,969::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 40 seconds elapsed, 42% of data processed, total data: 16456MB, processed data: 2040MB, remaining data: 9685MB, transfer speed 52MBps, zero pages: 1214867MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:38:52,973::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 50 seconds elapsed, 46% of data processed, total data: 16456MB, processed data: 2560MB, remaining data: 8944MB, transfer speed 52MBps, zero pages: 1271798MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:02,976::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 60 seconds elapsed, 51% of data processed, total data: 16456MB, processed data: 3080MB, remaining data: 8148MB, transfer speed 52MBps, zero pages: 1342711MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:12,979::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 70 seconds elapsed, 56% of data processed, total data: 16456MB, processed data: 3600MB, remaining data: 7353MB, transfer speed 52MBps, zero pages: 1413447MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:22,984::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 80 seconds elapsed, 59% of data processed, total data: 16456MB, processed data: 4120MB, remaining data: 6755MB, transfer speed 52MBps, zero pages: 1433707MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:32,987::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 90 seconds elapsed, 64% of data processed, total data: 16456MB, processed data: 4641MB, remaining data: 6032MB, transfer speed 52MBps, zero pages: 1486003MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:42,990::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 100 seconds elapsed, 67% of data processed, total data: 16456MB, processed data: 5161MB, remaining data: 5457MB, transfer speed 52MBps, zero pages: 1500433MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:39:52,993::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 110 seconds elapsed, 71% of data processed, total data: 16456MB, processed data: 5681MB, remaining data: 4836MB, transfer speed 52MBps, zero pages: 1526530MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:02,996::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 120 seconds elapsed, 75% of data processed, total data: 16456MB, processed data: 6201MB, remaining data: 4220MB, transfer speed 52MBps, zero pages: 1551477MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:12,999::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 130 seconds elapsed, 79% of data processed, total data: 16456MB, processed data: 6721MB, remaining data: 3574MB, transfer speed 52MBps, zero pages: 1583895MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:23,002::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 140 seconds elapsed, 82% of data processed, total data: 16456MB, processed data: 7242MB, remaining data: 2979MB, transfer speed 52MBps, zero pages: 1603409MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:33,005::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 150 seconds elapsed, 86% of data processed, total data: 16456MB, processed data: 7762MB, remaining data: 2382MB, transfer speed 52MBps, zero pages: 1623448MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:43,008::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 160 seconds elapsed, 90% of data processed, total data: 16456MB, processed data: 8282MB, remaining data: 1736MB, transfer speed 52MBps, zero pages: 1655832MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:40:53,011::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 170 seconds elapsed, 93% of data processed, total data: 16456MB, processed data: 8802MB, remaining data: 1166MB, transfer speed 52MBps, zero pages: 1668937MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:41:03,014::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 180 seconds elapsed, 97% of data processed, total data: 16456MB, processed data: 9322MB, remaining data: 581MB, transfer speed 52MBps, zero pages: 1685861MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>Thread-6320717::INFO::2017-03-24 11:41:13,019::migration::712::virt.vm::(monitor_migration) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Migration Progress: 190 seconds elapsed, 98% of data processed, total data: 16456MB, processed data: 9842MB, remaining data: 386MB, transfer speed 52MBps, zero pages: 1718676MB, compressed: 0MB, dirty rate: -1, memory iteration: -1<br>libvirtEventLoop::DEBUG::2017-03-24 11:41:21,007::vm::4291::virt.vm::(onLibvirtLifecycleEvent) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Suspended detail 0 opaque None<br>libvirtEventLoop::INFO::2017-03-24 11:41:21,025::vm::4815::virt.vm::(_logGuestCpuStatus) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU stopped: onSuspend<br>libvirtEventLoop::DEBUG::2017-03-24 11:41:21,069::vm::4291::virt.vm::(onLibvirtLifecycleEvent) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::event Resumed detail 0 opaque None<br>libvirtEventLoop::INFO::2017-03-24 11:41:21,069::vm::4815::virt.vm::(_logGuestCpuStatus) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::CPU running: onResume<br>Thread-6320715::DEBUG::2017-03-24 11:41:21,224::migration::715::virt.vm::(stop) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::stopping migration monitor thread<br>Thread-6320715::ERROR::2017-03-24 11:41:21,225::migration::252::virt.vm::(_recover) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::operation failed: migration job: unexpectedly failed<br>Thread-6320717::DEBUG::2017-03-24 11:41:21,226::migration::629::virt.vm::(run) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::stopped migration monitor thread<br>Thread-6320715::DEBUG::2017-03-24 11:41:21,257::guestagent::502::virt.vm::(send_lifecycle_event) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::send_lifecycle_event after_migration called<br>Thread-6320715::DEBUG::2017-03-24 11:41:21,284::__init__::209::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 19498739270, "4f627cc1-9b52-4eef-bf3a-c02e8a6303b8": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|4f627cc1-9b52-4eef-bf3a-c02e8a6303b8"}<br>Thread-6320715::ERROR::2017-03-24 11:41:21,285::migration::381::virt.vm::(run) vmId=`4f627cc1-9b52-4eef-bf3a-c02e8a6303b8`::Failed to migrate<br>jsonrpc.Executor/6::DEBUG::2017-03-24 11:41:29,161::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'VM.getMigrationStatus' in bridge with {u'vmID': u'4f627cc1-9b52-4eef-bf3a-c02e8a6303b8'}<br><br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">2017-03-24 11:54 GMT+01:00 Davide Ferrari <span dir="ltr"><<a href="mailto:davide@billymob.com" target="_blank">davide@billymob.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Helo<br><br></div>I have a VM on a host that seems it cannot be live-migrated away. I'm trying to migrate it to another (superior) cluster but please consider:<br></div>- VM isn't receiving too much tarffic and is not doing much at all<br></div>- I've already successfully live migrated other VMs from this host to the same other cluster<br><br></div>looking through engine.log I cannot see anything interesting a aprt from the generic ( I grepped for the job ID)<br><br>2017-03-24 10:30:52,186 INFO [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-42) [20518310] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: druid-co01., Source: vmhost04, Destination: vmhost06, User: admin@internal-authz).<br>2017-03-24 10:34:28,072 WARN [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-2) [6d6a3a53] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Failed to migrate VM druid-co01. to Host vmhost06 . Trying to migrate to another Host.<br>2017-03-24 10:34:28,676 INFO [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-2) [6d6a3a53] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: druid-co01., Source: vmhost04, Destination: vmhost11, User: admin@internal-authz).<br>2017-03-24 10:37:59,097 WARN [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-40) [779e9773] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Failed to migrate VM druid-co01. to Host vmhost11 . Trying to migrate to another Host.<br>2017-03-24 10:38:00,626 INFO [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-40) [779e9773] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: druid-co01, Source: vmhost04, Destination: vmhost08, User: admin@internal-authz).<br>2017-03-24 10:41:29,441 WARN [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-42) [6f032492] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Failed to migrate VM druid-co01 to Host vmhost08 . Trying to migrate to another Host.<br>2017-03-24 10:41:29,488 ERROR [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-42) [6f032492] Correlation ID: 20518310, Job ID: 039f0694-3e05-4f93-993d-<wbr>9e7383047873, Call Stack: null, Custom Event ID: -1, Message: Migration failed (VM: druid-co01, Source: vmhost04).<span class="HOEnZb"><font color="#888888"><br><br><div><div><div><div><div>-- <br><div class="m_-3670113318628776464gmail_signature"><div dir="ltr"><div>Davide Ferrari<br></div>Senior Systems Engineer<br></div></div>
</div></div></div></div></div></font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Davide Ferrari<br></div>Senior Systems Engineer<br></div></div>
</div>