<br><br><div class="gmail_quote">On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <span dir="ltr"><<a href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38 PM, Rami
Vaknin <span dir="ltr"><<a href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div> On 04/17/2012 05:36 PM, Martin Hovmöller
wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On Tue, Apr 17, 2012 at 2:55
PM, Rami Vaknin <span dir="ltr"><<a href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div> On 04/17/2012 03:45 PM, Martin Hovmöller
wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On Tue, Apr 17,
2012 at 2:11 PM, Itamar Heim <span dir="ltr"><<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 04/17/2012 12:47 PM, Martin
Hovmöller wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div> I can't live migrate the vm's
in my cluster:<br>
<br>
012-04-17 04:06:39,728 INFO<br>
[org.ovirt.engine.core.bll.MigrateVmCommand]
(pool-5-thread-48)<br>
Running command: MigrateVmCommand
internal: false. Entities affected
:<br>
ID:
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
Type: VM<br>
2012-04-17 04:06:39,743 INFO<br>
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(pool-5-thread-48)<br>
START, MigrateVDSCommand(vdsId =
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
</div>
dstHost=<a href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
<div><br>
migrationMethod=ONLINE), log id:
7c7201be<br>
2012-04-17 04:06:39,748 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48)
VdsBroker::migrate::Entered<br>
(vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
</div>
dstHost=<a href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
method=online
<div><br>
2012-04-17 04:06:39,749 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48) START,
MigrateBrokerVDSCommand(vdsId =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
</div>
dstHost=<a href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
<div>
<div><br>
migrationMethod=ONLINE), log id:
3c40c2de<br>
2012-04-17 04:06:39,834 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48) FINISH,
MigrateBrokerVDSCommand, log id:
3c40c2de<br>
2012-04-17 04:06:39,840 INFO<br>
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(pool-5-thread-48)<br>
FINISH, MigrateVDSCommand,
return: MigratingFrom, log id:
7c7201be<br>
2012-04-17 04:06:42,166 INFO<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-18)
vds::refreshVmList vm id<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
is migrating to vds
rhevh1.<domain><br>
ignoring it in the refresh till
migration is done<br>
[...]<br>
2012-04-17 04:07:33,852 INFO<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-34)
vds::refreshVmList vm id<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
status = Paused on vds<br>
rhevh1.<domain> ignoring
it in the refresh till migration
is done<br>
2012-04-17 04:07:37,407 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-71)
Rerun vm<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a.
Called from vds
rhevh2.<domain><br>
2012-04-17 04:07:37,414 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
(pool-5-thread-47) START,
MigrateStatusVDSCommand(vdsId =<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a),
log id: 8dfce1b<br>
2012-04-17 04:07:37,496 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47) Failed in
MigrateStatusVDS method<br>
2012-04-17 04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47) Error code
createErr and error message<br>
VDSGenericException:
VDSErrorException: Failed to
MigrateStatusVDS,<br>
error = Error creating the
requested virtual machine<br>
2012-04-17 04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47) Command<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return<br>
value<br>
Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br>
mStatus
Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br>
mCode 9<br>
mMessage
Error creating the requested
virtual machine<br>
<br>
<br>
2012-04-17 04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47) Vds:
rhevh2.<domain><br>
2012-04-17 04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VDSCommandBase]
(pool-5-thread-47)<br>
Command MigrateStatusVDS
execution failed. Exception:
VDSErrorException:<br>
VDSGenericException:
VDSErrorException: Failed to
MigrateStatusVDS,<br>
error = Error creating the
requested virtual machine<br>
2012-04-17 04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
(pool-5-thread-47) FINISH,
MigrateStatusVDSCommand, log id:
8dfce1b<br>
2012-04-17 04:07:37,528 WARN<br>
[org.ovirt.engine.core.bll.MigrateVmCommand]
(pool-5-thread-47)<br>
CanDoAction of action MigrateVm
failed.<br>
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br>
<br>
<br>
There is no problem regarding
storage or anything like that.
If I shut<br>
down the vm I can start it on
the other host without any
problems<br>
whatsoever.<br>
Do I need to do something to
make live migration work? Trying
to dig in<br>
the logs on the hypervisors, but
there's so much stuff being
logged there...<br>
<br>
<br>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
</blockquote>
<br>
vdsm log from the host that failed to
create the VM?<br>
</blockquote>
</div>
<br>
<div>The vdsm log is frighteningly long,
so I put it in pastebin: <a href="http://pastebin.com/BEeqFCCb" target="_blank">http://pastebin.com/BEeqFCCb</a></div>
<div>I don't really see anything
worthwhile myself, but on the other hand
I'm not sure what I'm looking for, so
it's mostly a massive wall of text to
me, but it should be somewhere in that
time frame.</div>
</blockquote>
</div>
</div>
Here is the failure, "code 9" is not too
informative, can you attach the libvirtd.log
file too?<br>
<br>
Thread-49287::DEBUG::2012-04-17
14:25:39,834::clientIF::81::vds::(wrapper)
return migrationCreate with {'status':
{'message': 'Error creating the requested
virtual machine', 'code': 9}}
<div><br>
<blockquote type="cite"> <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
</div>
<span><font color="#888888">
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</font></span></div>
</blockquote>
</div>
<div><br>
</div>
<div>I think this is the relevant info from the
libvirt.log:</div>
<div>
<div>2012-04-17 14:30:53.768+0000: 8506: debug :
virDomainLookupByUUIDString:1984 :
conn=0x7fa9a00bcde0,
uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
<div>2012-04-17 14:30:53.768+0000: 8506: debug :
virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0,
uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
<div>2012-04-17 14:30:53.771+0000: 8506: error :
virNetClientProgramDispatchError:170 : Domain not
found: no domain with matching uuid
'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div>
</div>
<div><br>
</div>
<div>What domain is this, is it the storage domain?</div>
</blockquote>
<br>
</div>
</div>
Domain in libvirt is simply a vm.
<div><br>
<blockquote type="cite"> <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</div>
</div>
</blockquote>
</div>
<br>
<div>Alright, so what does this mean then? The machine I'm
migrating to can't see the vm?<br>
That doesn't make sense since I can manually start it on that
host..</div>
</blockquote>
<br></div></div>
On second look at the vdsm.log I see that disk preparation in the
destination host failed (timed out), try to take a look at vdsm.log
at the destination.<br>
In addition, for some reason getVmStats returns "Virtual machine
does not exist" right before the migration.<br>
<br>
<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br>
Thread-49287::DEBUG::2012-04-17
14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call
migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable':
'true',.....<br>
Thread-49287::DEBUG::2012-04-17
14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination:
waiting 31s for path preparation<br>
<br>
Thread-49287::DEBUG::2012-04-17
14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting
for path preparation<br>
<br>
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}<div class="im"><br>
<blockquote type="cite">
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</div></div>
</blockquote></div>The vdsm log is already included earlier in the conversation; <a href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeqFCCb</a><div>I don't really know where to go from here. :/</div>