<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 04/19/2012 05:17 PM, Martin Hovmöller wrote:
<blockquote
cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com"
type="cite"><br>
<br>
<div class="gmail_quote">On Wed, Apr 18, 2012 at 11:06 AM, Rami
Vaknin <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div class="h5"> On 04/18/2012 11:30 AM, Martin Hovmöller
wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38
PM, Rami Vaknin <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div> On 04/17/2012 05:36 PM, Martin Hovmöller
wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On Tue, Apr 17,
2012 at 2:55 PM, Rami Vaknin <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rvaknin@redhat.com"
target="_blank">rvaknin@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div> On 04/17/2012 03:45 PM,
Martin Hovmöller wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">On
Tue, Apr 17, 2012 at 2:11
PM, Itamar Heim <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:iheim@redhat.com"
target="_blank">iheim@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>On 04/17/2012 12:47
PM, Martin Hovmöller
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div> I can't live
migrate the vm's in my
cluster:<br>
<br>
012-04-17 04:06:39,728
INFO<br>
[org.ovirt.engine.core.bll.MigrateVmCommand]
(pool-5-thread-48)<br>
Running command:
MigrateVmCommand
internal: false.
Entities affected :<br>
ID:
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
Type: VM<br>
2012-04-17
04:06:39,743 INFO<br>
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(pool-5-thread-48)<br>
START,
MigrateVDSCommand(vdsId
=
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
</div>
dstHost=<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
<div><br>
migrationMethod=ONLINE),
log id: 7c7201be<br>
2012-04-17
04:06:39,748 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48)
VdsBroker::migrate::Entered<br>
(vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
</div>
dstHost=<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
method=online
<div><br>
2012-04-17
04:06:39,749 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48)
START,
MigrateBrokerVDSCommand(vdsId
=<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
srcHost=10.23.30.130,<br>
dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br>
</div>
dstHost=<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a>
<<a
moz-do-not-send="true"
href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>,
<div>
<div><br>
migrationMethod=ONLINE),
log id: 3c40c2de<br>
2012-04-17
04:06:39,834 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br>
(pool-5-thread-48)
FINISH,
MigrateBrokerVDSCommand,
log id: 3c40c2de<br>
2012-04-17
04:06:39,840 INFO<br>
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(pool-5-thread-48)<br>
FINISH,
MigrateVDSCommand,
return:
MigratingFrom, log
id: 7c7201be<br>
2012-04-17
04:06:42,166 INFO<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-18)
vds::refreshVmList
vm id<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
is migrating to vds
rhevh1.<domain><br>
ignoring it in the
refresh till
migration is done<br>
[...]<br>
2012-04-17
04:07:33,852 INFO<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-34)
vds::refreshVmList
vm id<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
status = Paused on
vds<br>
rhevh1.<domain>
ignoring it in the
refresh till
migration is done<br>
2012-04-17
04:07:37,407 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br>
(QuartzScheduler_Worker-71)
Rerun vm<br>
ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a.
Called from vds
rhevh2.<domain><br>
2012-04-17
04:07:37,414 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
(pool-5-thread-47)
START,
MigrateStatusVDSCommand(vdsId
=<br>
f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br>
vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a),
log id: 8dfce1b<br>
2012-04-17
04:07:37,496 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47)
Failed in
MigrateStatusVDS
method<br>
2012-04-17
04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47)
Error code createErr
and error message<br>
VDSGenericException:
VDSErrorException:
Failed to
MigrateStatusVDS,<br>
error = Error
creating the
requested virtual
machine<br>
2012-04-17
04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47)
Command<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return<br>
value<br>
Class Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br>
mStatus
Class
Name:<br>
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br>
mCode
9<br>
mMessage
Error
creating the
requested virtual
machine<br>
<br>
<br>
2012-04-17
04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br>
(pool-5-thread-47)
Vds:
rhevh2.<domain><br>
2012-04-17
04:07:37,497 ERROR<br>
[org.ovirt.engine.core.vdsbroker.VDSCommandBase]
(pool-5-thread-47)<br>
Command
MigrateStatusVDS
execution failed.
Exception:
VDSErrorException:<br>
VDSGenericException:
VDSErrorException:
Failed to
MigrateStatusVDS,<br>
error = Error
creating the
requested virtual
machine<br>
2012-04-17
04:07:37,497 INFO<br>
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br>
(pool-5-thread-47)
FINISH,
MigrateStatusVDSCommand,
log id: 8dfce1b<br>
2012-04-17
04:07:37,528 WARN<br>
[org.ovirt.engine.core.bll.MigrateVmCommand]
(pool-5-thread-47)<br>
CanDoAction of
action MigrateVm
failed.<br>
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br>
<br>
<br>
There is no problem
regarding storage or
anything like that.
If I shut<br>
down the vm I can
start it on the
other host without
any problems<br>
whatsoever.<br>
Do I need to do
something to make
live migration work?
Trying to dig in<br>
the logs on the
hypervisors, but
there's so much
stuff being logged
there...<br>
<br>
<br>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a
moz-do-not-send="true"
href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a
moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
</blockquote>
<br>
vdsm log from the host
that failed to create the
VM?<br>
</blockquote>
</div>
<br>
<div>The vdsm log
is frighteningly long, so I
put it in pastebin: <a
moz-do-not-send="true"
href="http://pastebin.com/BEeqFCCb"
target="_blank">http://pastebin.com/BEeqFCCb</a></div>
<div>I don't really see
anything worthwhile myself,
but on the other hand I'm
not sure what I'm looking
for, so it's mostly a
massive wall of text to me,
but it should be somewhere
in that time frame.</div>
</blockquote>
</div>
</div>
Here is the failure, "code 9" is not
too informative, can you attach the
libvirtd.log file too?<br>
<br>
Thread-49287::DEBUG::2012-04-17
14:25:39,834::clientIF::81::vds::(wrapper)
return migrationCreate with
{'status': {'message': 'Error
creating the requested virtual
machine', 'code': 9}}
<div><br>
<blockquote type="cite"> <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
</div>
<span><font color="#888888">
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</font></span></div>
</blockquote>
</div>
<div><br>
</div>
<div>I think this is the relevant info
from the libvirt.log:</div>
<div>
<div>2012-04-17 14:30:53.768+0000: 8506:
debug :
virDomainLookupByUUIDString:1984 :
conn=0x7fa9a00bcde0,
uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
<div>2012-04-17 14:30:53.768+0000: 8506:
debug : virDomainLookupByUUID:1941 :
conn=0x7fa9a00bcde0,
uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div>
<div>2012-04-17 14:30:53.771+0000: 8506:
error :
virNetClientProgramDispatchError:170 :
Domain not found: no domain with
matching uuid
'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div>
</div>
<div><br>
</div>
<div>What domain is this, is it the
storage domain?</div>
</blockquote>
<br>
</div>
</div>
Domain in libvirt is simply a vm.
<div><br>
<blockquote type="cite"> <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</div>
</div>
</blockquote>
</div>
<br>
<div>Alright, so what does this mean then? The machine
I'm migrating to can't see the vm?<br>
That doesn't make sense since I can manually start
it on that host..</div>
</blockquote>
<br>
</div>
</div>
On second look at the vdsm.log I see that disk preparation
in the destination host failed (timed out), try to take a
look at vdsm.log at the destination.<br>
In addition, for some reason getVmStats returns "Virtual
machine does not exist" right before the migration.<br>
<br>
<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,054::clientIF::76::vds::(wrapper)
[10.23.30.130]::call getVmStats with
('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats
with {'status': {'message': 'Virtual machine does not
exist', 'code': 1}}<br>
Thread-49287::DEBUG::2012-04-17
14:25:07,381::clientIF::76::vds::(wrapper)
[10.23.30.130]::call migrationCreate with ({'bridge':
'ovirtmgmt', 'acpiEnable': 'true',.....<br>
Thread-49287::DEBUG::2012-04-17
14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration
destination: waiting 31s for path preparation<br>
<br>
Thread-49287::DEBUG::2012-04-17
14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while
waiting for path preparation<br>
<br>
14:25:07,054::clientIF::76::vds::(wrapper)
[10.23.30.130]::call getVmStats with
('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br>
Thread-49286::DEBUG::2012-04-17
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats
with {'status': {'message': 'Virtual machine does not
exist', 'code': 1}}
<div class="im"><br>
<blockquote type="cite"> <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</div>
</div>
</blockquote>
</div>
The vdsm log is already included earlier in the conversation; <a
moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeqFCCb</a>
<div>I don't really know where to go from here. :/</div>
</blockquote>
<br>
The exact issue that caused the prepareVolume to fail is not
mentioned in the attached log.<br>
<br>
Can you please tell whether the attached vdsm log is from the
migration source host or from the migration destination host, and
attach as well the second missing log? The vdsm log from the host
where the vm should run after the migration should contain a
failed/hanged lvm command on the vm's disk.<br>
<br>
<br>
<br>
<blockquote
cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com"
type="cite">
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
</pre>
</body>
</html>