<div dir="ltr">Hi<div><br></div><div>I specified ISO image and chose CD-ROM as first boot device.</div><div><br></div><div>Edgars</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 20, 2014 at 11:19 AM, Dafna Ron <span dir="ltr"><<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">when you booted the vm's for the first time, did you specify the ISO in vm's boot sequence or did you use run-once?<div>
<div class="h5"><br>
<br>
On 01/20/2014 09:53 AM, Edgars M. wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi again<br>
<br>
I tried to reproduce this issue and unfortunately I couldn't. So there must be something else, but I cannot figure out what is wrong, because it is happening only with some VMs. Nonetheless I attached new log files to investigate.<br>
<br>
And here is some more info:<br>
ovirt-engine-setup-3.3.2-1.<u></u>el6.noarch<br>
ovirt-engine-webadmin-portal-<u></u>3.3.2-1.el6.noarch<br>
ovirt-host-deploy-java-1.1.2-<u></u>1.el6.noarch<br>
ovirt-engine-restapi-3.3.2-1.<u></u>el6.noarch<br>
ovirt-image-uploader-3.3.1-1.<u></u>el6.noarch<br>
ovirt-release-el6-10-1.noarch<br>
ovirt-engine-userportal-3.3.2-<u></u>1.el6.noarch<br>
ovirt-engine-backend-3.3.2-1.<u></u>el6.noarch<br>
ovirt-engine-websocket-proxy-<u></u>3.3.2-1.el6.noarch<br>
ovirt-engine-dbscripts-3.3.2-<u></u>1.el6.noarch<br>
ovirt-log-collector-3.3.1-1.<u></u>el6.noarch<br>
ovirt-engine-sdk-python-3.3.0.<u></u>8-1.el6.noarch<br>
ovirt-engine-cli-3.3.0.6-1.<u></u>el6.noarch<br>
ovirt-engine-tools-3.3.2-1.<u></u>el6.noarch<br>
ovirt-iso-uploader-3.3.1-1.<u></u>el6.noarch<br>
ovirt-engine-lib-3.3.2-1.el6.<u></u>noarch<br>
ovirt-engine-3.3.2-1.el6.<u></u>noarch<br>
ovirt-host-deploy-1.1.2-1.el6.<u></u>noarch<br>
<br>
vdsm-xmlrpc-4.13.2-1.el6.<u></u>noarch<br>
vdsm-python-4.13.2-1.el6.x86_<u></u>64<br>
vdsm-python-cpopen-4.13.2-1.<u></u>el6.x86_64<br>
vdsm-4.13.2-1.el6.x86_64<br>
vdsm-cli-4.13.2-1.el6.noarch<br>
<br>
libvirt-lock-sanlock-0.10.2-<u></u>29.el6_5.2.x86_64<br>
libvirt-client-0.10.2-29.el6_<u></u>5.2.x86_64<br>
libvirt-python-0.10.2-29.el6_<u></u>5.2.x86_64<br>
libvirt-0.10.2-29.el6_5.2.x86_<u></u>64<br>
<br>
CentOS 6.5 x86_64 on both Engine and Nodes.<br>
<br>
SELinux disabled everywhere (I know, I know).<br>
There is no any snapshots for these VMs. These VMs were installed by booting from ISO image.<br>
There is nothing in libvirtd.log files<br>
<br>
<br>
BR<br>
Edgars<br>
<br>
<br>
<br></div></div><div class="im">
On Sat, Jan 18, 2014 at 6:39 PM, Dafna Ron <<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a> <mailto:<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>>> wrote:<br>
<br>
sounds like this is the issue to me... Edgars can you try to<br>
confirm that? :)<br>
Michal, was there a bug opened for this? I think that perhaps we<br>
should also add a clear error message - it would help debug this<br>
more easily.<br>
<br>
<br>
<br>
On 01/18/2014 05:34 PM, Michal Skrivanek wrote:<br>
<br>
<br>
On 18 Jan 2014, at 18:06, Itamar Heim <<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a><br></div><div><div class="h5">
<mailto:<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>>> wrote:<br>
<br>
On 01/18/2014 03:44 PM, Edgars M. wrote:<br>
Hi<br>
<br>
Thanks for your help. I will provide all log files a<br>
little bit later,<br>
but so far I have noticed some pattern when migration<br>
fails. Those<br>
particular VMs, which fails to migrate, were installed<br>
from ISO image,<br>
usual installation. So, I believe you can reproduce<br>
the issue by this:<br>
<br>
1. Upload ISO image to ISO domain<br>
2. Install new OS by booting from ISO image and check<br>
Attach CD in Boot<br>
Options<br>
3. Delete ISO image from ISO domain<br>
4. Try to migrate VM to another host.<br>
<br>
<br>
I have not tried this yet, but I have noticed that<br>
only those VMs fails<br>
which had been install from ISO image which is not in<br>
ISO domain<br>
anymore. I have also VMs created from templates and<br>
those VMs I can<br>
migrate just fine.<br>
<br>
did you stop the VMs post install? did you try to start<br>
them without the iso attached?<br>
otherwise, you can't start them, as there is a missing iso<br>
for the target qemu process<br>
<br>
Indeed. That is a known libvirt/qemu issue that even though<br>
the CD is defined as optional and VM can be started without it<br>
on original host, it fails upon migration when destination is<br>
being created.<br>
<br>
Thanks,<br>
michal<br>
<br>
I will provide more log files later.<br>
<br>
BR<br>
Edgars<br>
<br>
<br>
On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron<br>
<<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a> <mailto:<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>><br></div></div><div><div class="h5">
<mailto:<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a> <mailto:<a href="mailto:dron@redhat.com" target="_blank">dron@redhat.com</a>>>> wrote:<br>
<br>
I looked at the logs log and only vdsm from node2<br>
(which appears to<br>
be the src node) seems to have full info.<br>
<br>
Please attach the following complete logs:<br>
vdsm log from dst node<br>
engine log<br>
libvirt logs from both nodes.<br>
<br>
Also, can you please answer the following?<br>
libvirt and vdsm you are using?<br>
what is selinux status on both hosts<br>
(enforcing/permissive)?<br>
do you have snapshots on the vm or does it happen<br>
on a newly created<br>
disk?<br>
when you create the disk, is it from a template or<br>
is it a new image?<br>
<br>
Thanks,<br>
<br>
Dafna<br>
<br>
<br>
<br>
<br>
On 01/17/2014 10:21 PM, Itamar Heim wrote:<br>
<br>
On 01/18/2014 12:07 AM, Meital Bourvine wrote:<br>
<br>
I opened a bug about missing info in UI:<br>
<a href="https://bugzilla.redhat.com/__show_bug.cgi?id=1054994" target="_blank">https://bugzilla.redhat.com/__<u></u>show_bug.cgi?id=1054994</a><br>
<<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1054994" target="_blank">https://bugzilla.redhat.com/<u></u>show_bug.cgi?id=1054994</a>><br>
<br>
It actually failed with this error:<br>
Thread-1417::DEBUG::2014-01-17<br>
17:01:28,344::vm::768::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::starting<br>
migration monitor thread<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::__<u></u>libvirtconnection::108::__<u></u>libvirtconnection::(wrapper)<br>
Unknown libvirterror: ecode: 38 edom: 42<br>
level: 2 message:<br>
Failed to inquire lock: No such process<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::vm::745::vm.Vm::<u></u>__(cancel)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::canceling<br>
migration downtime thread<br>
Thread-1415::DEBUG::2014-01-17<br>
17:01:28,409::vm::815::vm.Vm::<u></u>__(stop)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::stopping<br>
migration monitor thread<br>
Thread-1416::DEBUG::2014-01-17<br>
17:01:28,409::vm::742::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::migration<br>
downtime thread exiting<br>
Thread-1415::ERROR::2014-01-17<br>
17:01:28,410::vm::238::vm.Vm::<u></u>__(_recover)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Failed to<br>
inquire lock: No such process<br>
Thread-1415::ERROR::2014-01-17<br>
17:01:28,619::vm::337::vm.Vm::<u></u>__(run)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Failed<br>
to migrate<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/vm.py", line<br>
323, in run<br>
self.___<u></u>startUnderlyingMigration()<br>
File "/usr/share/vdsm/vm.py", line 400, in<br>
_startUnderlyingMigration<br>
None, maxBandwidth)<br>
File "/usr/share/vdsm/vm.py", line<br>
838, in f<br>
ret = attr(*args, **kwargs)<br>
File<br>
"/usr/lib64/python2.6/site-__<u></u>packages/vdsm/__<u></u>libvirtconnection.py",<br>
line 76, in wrapper<br>
ret = f(*args, **kwargs)<br>
File<br>
"/usr/lib64/python2.6/site-__<u></u>packages/libvirt.py",<br>
line 1178, in migrateToURI2<br>
if ret == -1: raise libvirtError<br>
('virDomainMigrateToURI2() failed', dom=self)<br>
libvirtError: Failed to inquire lock: No<br>
such process<br>
Thread-26::ERROR::2014-01-17<br>
17:01:28,917::sampling::355::_<u></u>_vm.Vm::(collect)<br>
vmId=`b8787906-187a-4234-a0c9-<u></u>__58fc4ddf2a57`::Stats<br>
function failed: <AdvancedStatsFunction<br>
_highWrite at 0x26efb58><br>
<br>
The problem is that it doesn't say which<br>
process...<br>
<br>
<br>
this looks like noise post the migration<br>
failing, with probably<br>
a more relevant error in libvirt log<br>
<br>
<br>
----- Original Message -----<br>
<br>
From: "Itamar Heim" <<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a><br>
<mailto:<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>><br></div></div>
<mailto:<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a><div class="im"><br>
<mailto:<a href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>>>><br>
To: "Edgars M."<br>
<<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a> <mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>><br></div>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a><div class="im"><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>>>, "Meital Bourvine"<br>
<<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>><br></div>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a> <mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>>><br>
Cc: <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>> <mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><div class="im"><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>>, "Michal<br>
Skrivanek" <<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a><br>
<mailto:<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>><br></div>
<mailto:<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a><div><div class="h5"><br>
<mailto:<a href="mailto:mskrivan@redhat.com" target="_blank">mskrivan@redhat.com</a>>>><br>
Sent: Friday, January 17, 2014 9:47:11 PM<br>
Subject: Re: [Users] VM Migration failed<br>
<br>
On 01/17/2014 06:25 PM, Edgars M. wrote:<br>
<br>
Hi Meital<br>
<br>
I tried to migrate another VM and<br>
it also failed.<br>
<br>
This is what I get in UI:<br>
<br>
<br>
Migration started (VM: nophpapp01,<br>
Source:<br>
novmnode1, Destination:<br>
novmnode2, User: edgarsm).<br>
Migration failed due to Error:<br>
Fatal error during<br>
migration. Trying to<br>
migrate to another Host (VM:<br>
nophpapp01, Source:<br>
novmnode1, Destination:<br>
novmnode2).<br>
Migration failed due to Error:<br>
Fatal error during<br>
migration (VM:<br>
nophpapp01, Source: novmnode1,<br>
Destination: novmnode2).<br>
<br>
There is nothing in<br>
/var/log/messages, neither on<br>
engine server nor nodes.<br>
<br>
<br>
See attachments for engine and<br>
vdsm logs. (I am<br>
migrating from vmnode1<br>
to vmnode2)<br>
<br>
<br>
whatever the issue is here, can you<br>
please open a bug on<br>
trying to<br>
return more info to user on the<br>
migration error itself<br>
(for easier<br>
troubleshooting).<br>
<br>
thanks,<br>
Itamar<br>
<br>
<br>
Thanks for the help<br>
Edgars<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Jan 17, 2014 at 4:46 PM,<br>
Meital Bourvine<br>
<<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a> <mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a><br>
<mailto:<a href="mailto:mbourvin@redhat.com" target="_blank">mbourvin@redhat.com</a>>>><u></u>> wrote:<br>
<br>
Which error are you getting<br>
in the UI?<br>
Can you please attach the<br>
full engine and vdsm<br>
logs?<br>
Also, please check if there<br>
is a relevant<br>
error in /var/log/messages<br>
<br>
<br>
------------------------------<u></u>__----------------------------<u></u>--__------------<br>
<br>
<br>
*From: *"Edgars M."<br>
<<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.com</a><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>><br></div></div>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>><br>
<mailto:<a href="mailto:edgars.mazurs@gmail" target="_blank">edgars.mazurs@gmail</a>.<br>
<mailto:<a href="mailto:edgars.mazurs@gmail" target="_blank">edgars.mazurs@gmail</a>.>_<u></u>_com<br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a><div class="im"><br>
<mailto:<a href="mailto:edgars.mazurs@gmail.com" target="_blank">edgars.mazurs@gmail.<u></u>com</a>>>>><br>
*To: *<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br></div>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>> <mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><div><div class="h5"><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<mailto:<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>>><br>
*Sent: *Friday, January<br>
17, 2014 3:42:37 PM<br>
*Subject: *[Users] VM<br>
Migration failed<br>
<br>
<br>
Hi<br>
<br>
I am experiencing issues<br>
with manual VM<br>
migration. VM fails to<br>
migrate to other node in<br>
the same Cluster.<br>
Here are some<br>
relevant engine.log entries:<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.__<u></u>VdsUpdateRunTimeInfo]<br>
(DefaultQuartzScheduler___<u></u>Worker-73) Rerun vm<br>
a31cfd62-26fc-4396-8a83-__<u></u>1aed68c7fd39.<br>
Called from vds novmnode1<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Failed in<br>
MigrateStatusVDS method<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Error<br>
code migrateErr<br>
and error message<br>
VDSGenericException:<br>
VDSErrorException:<br>
Failed to<br>
MigrateStatusVDS, error<br>
= Fatal error<br>
during migration<br>
<br>
ERROR<br>
[org.ovirt.engine.core.__<u></u>vdsbroker.vdsbroker.__<u></u>MigrateStatusVDSCommand]<br>
(pool-6-thread-49) Command<br>
MigrateStatusVDS execution failed.<br>
Exception:<br>
VDSErrorException:<br>
VDSGenericException:<br>
VDSErrorException: Failed to<br>
MigrateStatusVDS, error = Fatal<br>
error during migration<br>
<br>
Both Engine and Node are<br>
running CentOS<br>
6.5 x64. oVirt Engine<br>
Version: 3.3.2-1.el6.<br>
VDSM version<br>
4.13.2-1. I restarted engine<br>
and vdsm, that did not help.<br>
<br>
Also in vdsm log file I<br>
see the following<br>
errors related to the<br>
same VM ID:<br>
<br>
Thread-27::ERROR::2014-01-17<br>
<br>
16:37:06,271::sampling::355::_<u></u>_vm.Vm::(collect)<br>
<br>
vmId=`a31cfd62-26fc-4396-8a83-<u></u>__1aed68c7fd39`::Stats<br>
function<br>
failed:<br>
<AdvancedStatsFunction _highWrite<br>
at 0x26efb58><br>
<br>
Any hints?<br>
<br>
BR<br>
Edgars<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
<mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>><br>
<a href="http://lists.ovirt.org/__mailman/listinfo/users" target="_blank">http://lists.ovirt.org/__<u></u>mailman/listinfo/users</a><br>
<<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<u></u>mailman/listinfo/users</a>><br>
<br>
<br>
<br></div></div><div class="im">
--<br>
Dafna Ron<br>
<br>
<br>
<br>
<br>
-- Dafna Ron<br>
<br>
<br>
</div></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Dafna Ron<br>
</font></span></blockquote></div><br></div>