Hi

Thanks for your help. I will provide all log files a little bit later, but so far I have noticed some pattern when migration fails. Those particular VMs, which fails to migrate, were installed from ISO image, usual installation. So, I believe you can reproduce the issue by this:

1. Upload ISO image to ISO domain
2. Install new OS by booting from ISO image and check Attach CD in Boot Options
3. Delete ISO image from ISO domain
4. Try to migrate VM to another host.


I have not tried this yet, but I have noticed that only those VMs fails which had been install from ISO image which is not in ISO domain anymore. I have also VMs created from templates and those VMs I can migrate just fine.

I will provide more log files later.

BR
Edgars


On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron <dron@redhat.com> wrote:
I looked at the logs log and only vdsm from node2 (which appears to be the src node) seems to have full info.

Please attach the following complete logs:
vdsm log from dst node
engine log
libvirt logs from both nodes.

Also, can you please answer the following?
libvirt and vdsm you are using?
what is selinux status on both hosts (enforcing/permissive)?
do you have snapshots on the vm or does it happen on a newly created disk?
when you create the disk, is it from a template or is it a new image?

Thanks,

Dafna




On 01/17/2014 10:21 PM, Itamar Heim wrote:
On 01/18/2014 12:07 AM, Meital Bourvine wrote:
I opened a bug about missing info in UI:
https://bugzilla.redhat.com/show_bug.cgi?id=1054994

It actually failed with this error:
Thread-1417::DEBUG::2014-01-17 17:01:28,344::vm::768::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::starting migration monitor thread
Thread-1415::DEBUG::2014-01-17 17:01:28,409::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to inquire lock: No such process
Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::745::vm.Vm::(cancel) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::canceling migration downtime thread
Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::815::vm.Vm::(stop) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::stopping migration monitor thread
Thread-1416::DEBUG::2014-01-17 17:01:28,409::vm::742::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::migration downtime thread exiting
Thread-1415::ERROR::2014-01-17 17:01:28,410::vm::238::vm.Vm::(_recover) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Failed to inquire lock: No such process
Thread-1415::ERROR::2014-01-17 17:01:28,619::vm::337::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Failed to migrate
Traceback (most recent call last):
   File "/usr/share/vdsm/vm.py", line 323, in run
     self._startUnderlyingMigration()
   File "/usr/share/vdsm/vm.py", line 400, in _startUnderlyingMigration
     None, maxBandwidth)
   File "/usr/share/vdsm/vm.py", line 838, in f
     ret = attr(*args, **kwargs)
   File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
     ret = f(*args, **kwargs)
   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
     if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Failed to inquire lock: No such process
Thread-26::ERROR::2014-01-17 17:01:28,917::sampling::355::vm.Vm::(collect) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x26efb58>

The problem is that it doesn't say which process...

this looks like noise post the migration failing, with probably a more relevant error in libvirt log


----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com>
To: "Edgars M." <edgars.mazurs@gmail.com>, "Meital Bourvine" <mbourvin@redhat.com>
Cc: users@ovirt.org, "Michal Skrivanek" <mskrivan@redhat.com>
Sent: Friday, January 17, 2014 9:47:11 PM
Subject: Re: [Users] VM Migration failed

On 01/17/2014 06:25 PM, Edgars M. wrote:
Hi Meital

I tried to migrate another VM and it also failed.

This is what I get in UI:


Migration started (VM: nophpapp01, Source: novmnode1, Destination:
novmnode2, User: edgarsm).
Migration failed due to Error: Fatal error during migration. Trying to
migrate to another Host (VM: nophpapp01, Source: novmnode1, Destination:
novmnode2).
Migration failed due to Error: Fatal error during migration (VM:
nophpapp01, Source: novmnode1, Destination: novmnode2).

There is nothing in /var/log/messages, neither on engine server nor nodes.


See attachments for engine and vdsm logs. (I am migrating from vmnode1
to vmnode2)

whatever the issue is here, can you please open a bug on trying to
return more info to user on the migration error itself (for easier
troubleshooting).

thanks,
     Itamar


Thanks for the help
Edgars







On Fri, Jan 17, 2014 at 4:46 PM, Meital Bourvine <mbourvin@redhat.com
<mailto:mbourvin@redhat.com>> wrote:

     Which error are you getting in the UI?
     Can you please attach the full engine and vdsm logs?
     Also, please check if there is a relevant error in /var/log/messages


------------------------------------------------------------------------

         *From: *"Edgars M." <edgars.mazurs@gmail.com
         <mailto:edgars.mazurs@gmail.com>>
         *To: *users@ovirt.org <mailto:users@ovirt.org>
         *Sent: *Friday, January 17, 2014 3:42:37 PM
         *Subject: *[Users] VM Migration failed


         Hi

         I am experiencing issues with manual VM migration. VM fails to
         migrate to other node in the same Cluster. Here are some
         relevant engine.log entries:

         ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
         (DefaultQuartzScheduler_Worker-73) Rerun vm
         a31cfd62-26fc-4396-8a83-1aed68c7fd39. Called from vds novmnode1

         ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
         (pool-6-thread-49) Failed in MigrateStatusVDS method

         ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
         (pool-6-thread-49) Error code migrateErr and error message
         VDSGenericException: VDSErrorException: Failed to
         MigrateStatusVDS, error = Fatal error during migration

         ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
         (pool-6-thread-49) Command MigrateStatusVDS execution failed.
         Exception: VDSErrorException: VDSGenericException:
         VDSErrorException: Failed to MigrateStatusVDS, error = Fatal
         error during migration

         Both Engine and Node are running CentOS 6.5 x64. oVirt Engine
         Version: 3.3.2-1.el6. VDSM version 4.13.2-1. I restarted engine
         and vdsm, that did not help.

         Also in vdsm log file I see the following errors related to the
         same VM ID:

         Thread-27::ERROR::2014-01-17
         16:37:06,271::sampling::355::vm.Vm::(collect)
         vmId=`a31cfd62-26fc-4396-8a83-1aed68c7fd39`::Stats function
         failed: <AdvancedStatsFunction _highWrite at 0x26efb58>

         Any hints?

         BR
         Edgars

         _______________________________________________
         Users mailing list
         Users@ovirt.org <mailto:Users@ovirt.org>
         http://lists.ovirt.org/mailman/listinfo/users





_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users





_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


--
Dafna Ron