<div dir="ltr">Hi<div><br></div><div>Thanks for your help. I will provide all log files a little bit later, but so far I have noticed some pattern when migration fails. Those particular VMs, which fails to migrate, were installed from ISO image, usual installation. So, I believe you can reproduce the issue by this:</div>
<div><br></div><div>1. Upload ISO image to ISO domain</div><div>2. Install new OS by booting from ISO image and check Attach CD in Boot Options</div><div>3. Delete ISO image from ISO domain</div><div>4. Try to migrate VM to another host.</div>
<div><br></div><div><br></div><div>I have not tried this yet, but I have noticed that only those VMs fails which had been install from ISO image which is not in ISO domain anymore. I have also VMs created from templates and those VMs I can migrate just fine.</div>
<div><br></div><div>I will provide more log files later.</div><div><br></div><div>BR</div><div>Edgars</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron <span dir="ltr"><<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I looked at the logs log and only vdsm from node2 (which appears to be the src node) seems to have full info.<br>
<br>
Please attach the following complete logs:<br>
vdsm log from dst node<br>
engine log<br>
libvirt logs from both nodes.<br>
<br>
Also, can you please answer the following?<br>
libvirt and vdsm you are using?<br>
what is selinux status on both hosts (enforcing/permissive)?<br>
do you have snapshots on the vm or does it happen on a newly created disk?<br>
when you create the disk, is it from a template or is it a new image?<br>
<br>
Thanks,<br>
<br>
Dafna<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
On 01/17/2014 10:21 PM, Itamar Heim wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 01/18/2014 12:07 AM, Meital Bourvine wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I opened a bug about missing info in UI:<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1054994" target="_blank">https://bugzilla.redhat.com/<u></u>show_bug.cgi?id=1054994</a><br>
<br>
It actually failed with this error:<br>
Thread-1417::DEBUG::2014-01-17 17:01:28,344::vm::768::vm.Vm::<u></u>(run) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::starting migration monitor thread<br>
Thread-1415::DEBUG::2014-01-17 17:01:28,409::<u></u>libvirtconnection::108::<u></u>libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to inquire lock: No such process<br>
Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::745::vm.Vm::<u></u>(cancel) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::canceling migration downtime thread<br>
Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::815::vm.Vm::<u></u>(stop) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::stopping migration monitor thread<br>
Thread-1416::DEBUG::2014-01-17 17:01:28,409::vm::742::vm.Vm::<u></u>(run) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::migration downtime thread exiting<br>
Thread-1415::ERROR::2014-01-17 17:01:28,410::vm::238::vm.Vm::<u></u>(_recover) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::Failed to inquire lock: No such process<br>
Thread-1415::ERROR::2014-01-17 17:01:28,619::vm::337::vm.Vm::<u></u>(run) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::Failed to migrate<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/vm.py", line 323, in run<br>
self._<u></u>startUnderlyingMigration()<br>
File "/usr/share/vdsm/vm.py", line 400, in _startUnderlyingMigration<br>
None, maxBandwidth)<br>
File "/usr/share/vdsm/vm.py", line 838, in f<br>
ret = attr(*args, **kwargs)<br>
File "/usr/lib64/python2.6/site-<u></u>packages/vdsm/<u></u>libvirtconnection.py", line 76, in wrapper<br>
ret = f(*args, **kwargs)<br>
File "/usr/lib64/python2.6/site-<u></u>packages/libvirt.py", line 1178, in migrateToURI2<br>
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)<br>
libvirtError: Failed to inquire lock: No such process<br>
Thread-26::ERROR::2014-01-17 17:01:28,917::sampling::355::<u></u>vm.Vm::(collect) vmId=`b8787906-187a-4234-a0c9-<u></u>58fc4ddf2a57`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x26efb58><br>
<br>
The problem is that it doesn't say which process...<br>
</blockquote>
<br>
this looks like noise post the migration failing, with probably a more relevant error in libvirt log<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
----- Original Message -----<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
From: "Itamar Heim" <<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>><br>
To: "Edgars M." <<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a>>, "Meital Bourvine" <<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>><br>
Cc: <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>, "Michal Skrivanek" <<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>><br>
Sent: Friday, January 17, 2014 9:47:11 PM<br>
Subject: Re: [Users] VM Migration failed<br>
<br>
On 01/17/2014 06:25 PM, Edgars M. wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Meital<br>
<br>
I tried to migrate another VM and it also failed.<br>
<br>
This is what I get in UI:<br>
<br>
<br>
Migration started (VM: nophpapp01, Source: novmnode1, Destination:<br>
novmnode2, User: edgarsm).<br>
Migration failed due to Error: Fatal error during migration. Trying to<br>
migrate to another Host (VM: nophpapp01, Source: novmnode1, Destination:<br>
novmnode2).<br>
Migration failed due to Error: Fatal error during migration (VM:<br>
nophpapp01, Source: novmnode1, Destination: novmnode2).<br>
<br>
There is nothing in /var/log/messages, neither on engine server nor nodes.<br>
<br>
<br>
See attachments for engine and vdsm logs. (I am migrating from vmnode1<br>
to vmnode2)<br>
</blockquote>
<br>
whatever the issue is here, can you please open a bug on trying to<br>
return more info to user on the migration error itself (for easier<br>
troubleshooting).<br>
<br>
thanks,<br>
Itamar<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks for the help<br>
Edgars<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Jan 17, 2014 at 4:46 PM, Meital Bourvine <<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>> wrote:<br>
<br>
Which error are you getting in the UI?<br>
Can you please attach the full engine and vdsm logs?<br>
Also, please check if there is a relevant error in /var/log/messages<br>
<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------ <br>
<br>
*From: *"Edgars M." <<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>><br>
*To: *<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a> <mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
*Sent: *Friday, January 17, 2014 3:42:37 PM<br>
*Subject: *[Users] VM Migration failed<br>
<br>
<br>
Hi<br>
<br>
I am experiencing issues with manual VM migration. VM fails to<br>
migrate to other node in the same Cluster. Here are some<br>
relevant engine.log entries:<br>
<br>
ERROR [org.ovirt.engine.core.<u></u>vdsbroker.<u></u>VdsUpdateRunTimeInfo]<br>
(DefaultQuartzScheduler_<u></u>Worker-73) Rerun vm<br>
a31cfd62-26fc-4396-8a83-<u></u>1aed68c7fd39. Called from vds novmnode1<br>
<br>
ERROR<br>
[org.ovirt.engine.core.<u></u>vdsbroker.vdsbroker.<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Failed in MigrateStatusVDS method<br>
<br>
ERROR<br>
[org.ovirt.engine.core.<u></u>vdsbroker.vdsbroker.<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Error code migrateErr and error message<br>
VDSGenericException: VDSErrorException: Failed to<br>
MigrateStatusVDS, error = Fatal error during migration<br>
<br>
ERROR<br>
[org.ovirt.engine.core.<u></u>vdsbroker.vdsbroker.<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Command MigrateStatusVDS execution failed.<br>
Exception: VDSErrorException: VDSGenericException:<br>
VDSErrorException: Failed to MigrateStatusVDS, error = Fatal<br>
error during migration<br>
<br>
Both Engine and Node are running CentOS 6.5 x64. oVirt Engine<br>
Version: 3.3.2-1.el6. VDSM version 4.13.2-1. I restarted engine<br>
and vdsm, that did not help.<br>
<br>
Also in vdsm log file I see the following errors related to the<br>
same VM ID:<br>
<br>
Thread-27::ERROR::2014-01-17<br>
16:37:06,271::sampling::355::<u></u>vm.Vm::(collect)<br>
vmId=`a31cfd62-26fc-4396-8a83-<u></u>1aed68c7fd39`::Stats function<br>
failed: <AdvancedStatsFunction _highWrite at 0x26efb58><br>
<br>
Any hints?<br>
<br>
BR<br>
Edgars<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
<br>
</blockquote></blockquote>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a><br>
</blockquote>
<br>
<br></div></div><span class="HOEnZb"><font color="#888888">
-- <br>
Dafna Ron<br>
</font></span></blockquote></div><br></div>