<div dir="ltr">Hi again<div><br></div><div>I tried to reproduce this issue and unfortunately I couldn't. So there must be something else, but I cannot figure out what is wrong, because it is happening only with some VMs. Nonetheless I attached new log files to investigate. </div>
<div><br></div><div>And here is some more info:</div><div><div>ovirt-engine-setup-3.3.2-1.el6.noarch</div><div>ovirt-engine-webadmin-portal-3.3.2-1.el6.noarch</div><div>ovirt-host-deploy-java-1.1.2-1.el6.noarch</div><div>
ovirt-engine-restapi-3.3.2-1.el6.noarch</div><div>ovirt-image-uploader-3.3.1-1.el6.noarch</div><div>ovirt-release-el6-10-1.noarch</div><div>ovirt-engine-userportal-3.3.2-1.el6.noarch</div><div>ovirt-engine-backend-3.3.2-1.el6.noarch</div>
<div>ovirt-engine-websocket-proxy-3.3.2-1.el6.noarch</div><div>ovirt-engine-dbscripts-3.3.2-1.el6.noarch</div><div>ovirt-log-collector-3.3.1-1.el6.noarch</div><div>ovirt-engine-sdk-python-3.3.0.8-1.el6.noarch</div><div>ovirt-engine-cli-3.3.0.6-1.el6.noarch</div>
<div>ovirt-engine-tools-3.3.2-1.el6.noarch</div><div>ovirt-iso-uploader-3.3.1-1.el6.noarch</div><div>ovirt-engine-lib-3.3.2-1.el6.noarch</div><div>ovirt-engine-3.3.2-1.el6.noarch</div><div>ovirt-host-deploy-1.1.2-1.el6.noarch</div>
</div><div><br></div><div><div>vdsm-xmlrpc-4.13.2-1.el6.noarch</div><div>vdsm-python-4.13.2-1.el6.x86_64</div><div>vdsm-python-cpopen-4.13.2-1.el6.x86_64</div><div>vdsm-4.13.2-1.el6.x86_64</div><div>vdsm-cli-4.13.2-1.el6.noarch</div>
</div><div><br></div><div><div>libvirt-lock-sanlock-0.10.2-29.el6_5.2.x86_64</div><div>libvirt-client-0.10.2-29.el6_5.2.x86_64</div><div>libvirt-python-0.10.2-29.el6_5.2.x86_64</div><div>libvirt-0.10.2-29.el6_5.2.x86_64</div>
</div><div><br></div><div>CentOS 6.5 x86_64 on both Engine and Nodes.</div><div><br></div><div>SELinux disabled everywhere (I know, I know).</div><div>There is no any snapshots for these VMs. These VMs were installed by booting from ISO image.</div>
<div>There is nothing in libvirtd.log files</div><div><br></div><div><br></div><div>BR</div><div>Edgars</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jan 18, 2014 at 6:39 PM, Dafna Ron <span dir="ltr"><<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">sounds like this is the issue to me... Edgars can you try to confirm that? :)<br>
Michal, was there a bug opened for this? I think that perhaps we should also add a clear error message - it would help debug this more easily.<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 01/18/2014 05:34 PM, Michal Skrivanek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 18 Jan 2014, at 18:06, Itamar Heim <<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 01/18/2014 03:44 PM, Edgars M. wrote:<br>
Hi<br>
<br>
Thanks for your help. I will provide all log files a little bit later,<br>
but so far I have noticed some pattern when migration fails. Those<br>
particular VMs, which fails to migrate, were installed from ISO image,<br>
usual installation. So, I believe you can reproduce the issue by this:<br>
<br>
1. Upload ISO image to ISO domain<br>
2. Install new OS by booting from ISO image and check Attach CD in Boot<br>
Options<br>
3. Delete ISO image from ISO domain<br>
4. Try to migrate VM to another host.<br>
<br>
<br>
I have not tried this yet, but I have noticed that only those VMs fails<br>
which had been install from ISO image which is not in ISO domain<br>
anymore. I have also VMs created from templates and those VMs I can<br>
migrate just fine.<br>
</blockquote>
did you stop the VMs post install? did you try to start them without the iso attached?<br>
otherwise, you can't start them, as there is a missing iso for the target qemu process<br>
</blockquote>
Indeed. That is a known libvirt/qemu issue that even though the CD is defined as optional and VM can be started without it on original host, it fails upon migration when destination is being created.<br>
<br>
Thanks,<br>
michal<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I will provide more log files later.<br>
<br>
BR<br>
Edgars<br>
<br>
<br>
On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron <<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a><br>
<mailto:<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>>> wrote:<br>
<br>
I looked at the logs log and only vdsm from node2 (which appears to<br>
be the src node) seems to have full info.<br>
<br>
Please attach the following complete logs:<br>
vdsm log from dst node<br>
engine log<br>
libvirt logs from both nodes.<br>
<br>
Also, can you please answer the following?<br>
libvirt and vdsm you are using?<br>
what is selinux status on both hosts (enforcing/permissive)?<br>
do you have snapshots on the vm or does it happen on a newly created<br>
disk?<br>
when you create the disk, is it from a template or is it a new image?<br>
<br>
Thanks,<br>
<br>
Dafna<br>
<br>
<br>
<br>
<br>
On 01/17/2014 10:21 PM, Itamar Heim wrote:<br>
<br>
On 01/18/2014 12:07 AM, Meital Bourvine wrote:<br>
<br>
I opened a bug about missing info in UI:<br>
<a href="https://bugzilla.redhat.com/__show_bug.cgi?id=1054994" target="_blank">https://bugzilla.redhat.com/__<u></u>show_bug.cgi?id=1054994</a><br>
<<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1054994" target="_blank">https://bugzilla.redhat.com/<u></u>show_bug.cgi?id=1054994</a>><br>
<br>
It actually failed with this error:<br>
Thread-1417::DEBUG::2014-01-17<br>
17:01:28,344::vm::768::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::starting<br>
migration monitor thread<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::__<u></u>libvirtconnection::108::__<u></u>libvirtconnection::(wrapper)<br>
Unknown libvirterror: ecode: 38 edom: 42 level: 2 message:<br>
Failed to inquire lock: No such process<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::vm::745::vm.Vm::<u></u>__(cancel)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::canceling<br>
migration downtime thread<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::vm::815::vm.Vm::<u></u>__(stop)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::stopping<br>
migration monitor thread<br>
Thread-1416::DEBUG::2014-01-17<br>
17:01:28,409::vm::742::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::migration<br>
downtime thread exiting<br>
Thread-1415::ERROR::2014-01-17<br>
17:01:28,410::vm::238::vm.Vm::<u></u>__(_recover)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Failed to<br>
inquire lock: No such process<br>
Thread-1415::ERROR::2014-01-17<br>
17:01:28,619::vm::337::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Failed to migrate<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/vm.py", line 323, in run<br>
self.___<u></u>startUnderlyingMigration()<br>
File "/usr/share/vdsm/vm.py", line 400, in<br>
_startUnderlyingMigration<br>
None, maxBandwidth)<br>
File "/usr/share/vdsm/vm.py", line 838, in f<br>
ret = attr(*args, **kwargs)<br>
File<br>
"/usr/lib64/python2.6/site-__<u></u>packages/vdsm/__<u></u>libvirtconnection.py",<br>
line 76, in wrapper<br>
ret = f(*args, **kwargs)<br>
File "/usr/lib64/python2.6/site-__<u></u>packages/libvirt.py",<br>
line 1178, in migrateToURI2<br>
if ret == -1: raise libvirtError<br>
('virDomainMigrateToURI2() failed', dom=self)<br>
libvirtError: Failed to inquire lock: No such process<br>
Thread-26::ERROR::2014-01-17<br>
17:01:28,917::sampling::355::_<u></u>_vm.Vm::(collect)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Stats<br>
function failed: <AdvancedStatsFunction _highWrite at 0x26efb58><br>
<br>
The problem is that it doesn't say which process...<br>
<br>
<br>
this looks like noise post the migration failing, with probably<br>
a more relevant error in libvirt log<br>
<br>
<br>
----- Original Message -----<br>
<br>
From: "Itamar Heim" <<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a><br>
<mailto:<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>>><br>
To: "Edgars M." <<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>>, "Meital Bourvine"<br>
<<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a> <mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>><br>
Cc: <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a> <mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>, "Michal<br>
Skrivanek" <<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a><br>
<mailto:<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>>><br>
Sent: Friday, January 17, 2014 9:47:11 PM<br>
Subject: Re: [Users] VM Migration failed<br>
<br>
On 01/17/2014 06:25 PM, Edgars M. wrote:<br>
<br>
Hi Meital<br>
<br>
I tried to migrate another VM and it also failed.<br>
<br>
This is what I get in UI:<br>
<br>
<br>
Migration started (VM: nophpapp01, Source:<br>
novmnode1, Destination:<br>
novmnode2, User: edgarsm).<br>
Migration failed due to Error: Fatal error during<br>
migration. Trying to<br>
migrate to another Host (VM: nophpapp01, Source:<br>
novmnode1, Destination:<br>
novmnode2).<br>
Migration failed due to Error: Fatal error during<br>
migration (VM:<br>
nophpapp01, Source: novmnode1, Destination: novmnode2).<br>
<br>
There is nothing in /var/log/messages, neither on<br>
engine server nor nodes.<br>
<br>
<br>
See attachments for engine and vdsm logs. (I am<br>
migrating from vmnode1<br>
to vmnode2)<br>
<br>
<br>
whatever the issue is here, can you please open a bug on<br>
trying to<br>
return more info to user on the migration error itself<br>
(for easier<br>
troubleshooting).<br>
<br>
thanks,<br>
Itamar<br>
<br>
<br>
Thanks for the help<br>
Edgars<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Jan 17, 2014 at 4:46 PM, Meital Bourvine<br>
<<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a> <mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>>> wrote:<br>
<br>
Which error are you getting in the UI?<br>
Can you please attach the full engine and vdsm<br>
logs?<br>
Also, please check if there is a relevant<br>
error in /var/log/messages<br>
<br>
<br>
------------------------------<u></u>__----------------------------<u></u>--__------------<br>
<br>
<br>
*From: *"Edgars M."<br>
<<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>><br>
<mailto:<a href="mailto:edgars.mazurs@gmail." target="_blank">edgars.mazurs@gmail.</a>__<u></u>com<br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>>><br>
*To: *<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>> <mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>><br>
*Sent: *Friday, January 17, 2014 3:42:37 PM<br>
*Subject: *[Users] VM Migration failed<br>
<br>
<br>
Hi<br>
<br>
I am experiencing issues with manual VM<br>
migration. VM fails to<br>
migrate to other node in the same Cluster.<br>
Here are some<br>
relevant engine.log entries:<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.__<u></u>VdsUpdateRunTimeInfo]<br>
(DefaultQuartzScheduler___<u></u>Worker-73) Rerun vm<br>
a31cfd62-26fc-4396-8a83-__<u></u>1aed68c7fd39.<br>
Called from vds novmnode1<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Failed in<br>
MigrateStatusVDS method<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Error code migrateErr<br>
and error message<br>
VDSGenericException: VDSErrorException:<br>
Failed to<br>
MigrateStatusVDS, error = Fatal error<br>
during migration<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Command<br>
MigrateStatusVDS execution failed.<br>
Exception: VDSErrorException:<br>
VDSGenericException:<br>
VDSErrorException: Failed to<br>
MigrateStatusVDS, error = Fatal<br>
error during migration<br>
<br>
Both Engine and Node are running CentOS<br>
6.5 x64. oVirt Engine<br>
Version: 3.3.2-1.el6. VDSM version<br>
4.13.2-1. I restarted engine<br>
and vdsm, that did not help.<br>
<br>
Also in vdsm log file I see the following<br>
errors related to the<br>
same VM ID:<br>
<br>
Thread-27::ERROR::2014-01-17<br>
<br>
16:37:06,271::sampling::355::_<u></u>_vm.Vm::(collect)<br>
<br>
vmId=`a31cfd62-26fc-4396-8a83-<u></u>__1aed68c7fd39`::Stats function<br>
failed: <AdvancedStatsFunction _highWrite<br>
at 0x26efb58><br>
<br>
Any hints?<br>
<br>
BR<br>
Edgars<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br>
--<br>
Dafna Ron<br>
<br>
<br>
</blockquote></blockquote></blockquote>
<br>
<br></div></div><span class="HOEnZb"><font color="#888888">
-- <br>
Dafna Ron<br>
</font></span></blockquote></div><br></div>