[Users] VM Migration failed
Dafna Ron
dron at redhat.com
Mon Jan 20 10:19:32 UTC 2014
when you booted the vm's for the first time, did you specify the ISO in
vm's boot sequence or did you use run-once?
On 01/20/2014 09:53 AM, Edgars M. wrote:
> Hi again
>
> I tried to reproduce this issue and unfortunately I couldn't. So there
> must be something else, but I cannot figure out what is wrong, because
> it is happening only with some VMs. Nonetheless I attached new log
> files to investigate.
>
> And here is some more info:
> ovirt-engine-setup-3.3.2-1.el6.noarch
> ovirt-engine-webadmin-portal-3.3.2-1.el6.noarch
> ovirt-host-deploy-java-1.1.2-1.el6.noarch
> ovirt-engine-restapi-3.3.2-1.el6.noarch
> ovirt-image-uploader-3.3.1-1.el6.noarch
> ovirt-release-el6-10-1.noarch
> ovirt-engine-userportal-3.3.2-1.el6.noarch
> ovirt-engine-backend-3.3.2-1.el6.noarch
> ovirt-engine-websocket-proxy-3.3.2-1.el6.noarch
> ovirt-engine-dbscripts-3.3.2-1.el6.noarch
> ovirt-log-collector-3.3.1-1.el6.noarch
> ovirt-engine-sdk-python-3.3.0.8-1.el6.noarch
> ovirt-engine-cli-3.3.0.6-1.el6.noarch
> ovirt-engine-tools-3.3.2-1.el6.noarch
> ovirt-iso-uploader-3.3.1-1.el6.noarch
> ovirt-engine-lib-3.3.2-1.el6.noarch
> ovirt-engine-3.3.2-1.el6.noarch
> ovirt-host-deploy-1.1.2-1.el6.noarch
>
> vdsm-xmlrpc-4.13.2-1.el6.noarch
> vdsm-python-4.13.2-1.el6.x86_64
> vdsm-python-cpopen-4.13.2-1.el6.x86_64
> vdsm-4.13.2-1.el6.x86_64
> vdsm-cli-4.13.2-1.el6.noarch
>
> libvirt-lock-sanlock-0.10.2-29.el6_5.2.x86_64
> libvirt-client-0.10.2-29.el6_5.2.x86_64
> libvirt-python-0.10.2-29.el6_5.2.x86_64
> libvirt-0.10.2-29.el6_5.2.x86_64
>
> CentOS 6.5 x86_64 on both Engine and Nodes.
>
> SELinux disabled everywhere (I know, I know).
> There is no any snapshots for these VMs. These VMs were installed by
> booting from ISO image.
> There is nothing in libvirtd.log files
>
>
> BR
> Edgars
>
>
>
> On Sat, Jan 18, 2014 at 6:39 PM, Dafna Ron <dron at redhat.com
> <mailto:dron at redhat.com>> wrote:
>
> sounds like this is the issue to me... Edgars can you try to
> confirm that? :)
> Michal, was there a bug opened for this? I think that perhaps we
> should also add a clear error message - it would help debug this
> more easily.
>
>
>
> On 01/18/2014 05:34 PM, Michal Skrivanek wrote:
>
>
> On 18 Jan 2014, at 18:06, Itamar Heim <iheim at redhat.com
> <mailto:iheim at redhat.com>> wrote:
>
> On 01/18/2014 03:44 PM, Edgars M. wrote:
> Hi
>
> Thanks for your help. I will provide all log files a
> little bit later,
> but so far I have noticed some pattern when migration
> fails. Those
> particular VMs, which fails to migrate, were installed
> from ISO image,
> usual installation. So, I believe you can reproduce
> the issue by this:
>
> 1. Upload ISO image to ISO domain
> 2. Install new OS by booting from ISO image and check
> Attach CD in Boot
> Options
> 3. Delete ISO image from ISO domain
> 4. Try to migrate VM to another host.
>
>
> I have not tried this yet, but I have noticed that
> only those VMs fails
> which had been install from ISO image which is not in
> ISO domain
> anymore. I have also VMs created from templates and
> those VMs I can
> migrate just fine.
>
> did you stop the VMs post install? did you try to start
> them without the iso attached?
> otherwise, you can't start them, as there is a missing iso
> for the target qemu process
>
> Indeed. That is a known libvirt/qemu issue that even though
> the CD is defined as optional and VM can be started without it
> on original host, it fails upon migration when destination is
> being created.
>
> Thanks,
> michal
>
> I will provide more log files later.
>
> BR
> Edgars
>
>
> On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron
> <dron at redhat.com <mailto:dron at redhat.com>
> <mailto:dron at redhat.com <mailto:dron at redhat.com>>> wrote:
>
> I looked at the logs log and only vdsm from node2
> (which appears to
> be the src node) seems to have full info.
>
> Please attach the following complete logs:
> vdsm log from dst node
> engine log
> libvirt logs from both nodes.
>
> Also, can you please answer the following?
> libvirt and vdsm you are using?
> what is selinux status on both hosts
> (enforcing/permissive)?
> do you have snapshots on the vm or does it happen
> on a newly created
> disk?
> when you create the disk, is it from a template or
> is it a new image?
>
> Thanks,
>
> Dafna
>
>
>
>
> On 01/17/2014 10:21 PM, Itamar Heim wrote:
>
> On 01/18/2014 12:07 AM, Meital Bourvine wrote:
>
> I opened a bug about missing info in UI:
> https://bugzilla.redhat.com/__show_bug.cgi?id=1054994
>
> <https://bugzilla.redhat.com/show_bug.cgi?id=1054994>
>
> It actually failed with this error:
> Thread-1417::DEBUG::2014-01-17
> 17:01:28,344::vm::768::vm.Vm::__(run)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::starting
> migration monitor thread
> Thread-1415::DEBUG::2014-01-17
>
> 17:01:28,409::__libvirtconnection::108::__libvirtconnection::(wrapper)
> Unknown libvirterror: ecode: 38 edom: 42
> level: 2 message:
> Failed to inquire lock: No such process
> Thread-1415::DEBUG::2014-01-17
> 17:01:28,409::vm::745::vm.Vm::__(cancel)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::canceling
> migration downtime thread
> Thread-1415::DEBUG::2014-01-17
> 17:01:28,409::vm::815::vm.Vm::__(stop)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::stopping
> migration monitor thread
> Thread-1416::DEBUG::2014-01-17
> 17:01:28,409::vm::742::vm.Vm::__(run)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::migration
> downtime thread exiting
> Thread-1415::ERROR::2014-01-17
> 17:01:28,410::vm::238::vm.Vm::__(_recover)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Failed to
> inquire lock: No such process
> Thread-1415::ERROR::2014-01-17
> 17:01:28,619::vm::337::vm.Vm::__(run)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Failed
> to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line
> 323, in run
> self.___startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 400, in
> _startUnderlyingMigration
> None, maxBandwidth)
> File "/usr/share/vdsm/vm.py", line
> 838, in f
> ret = attr(*args, **kwargs)
> File
>
> "/usr/lib64/python2.6/site-__packages/vdsm/__libvirtconnection.py",
> line 76, in wrapper
> ret = f(*args, **kwargs)
> File
> "/usr/lib64/python2.6/site-__packages/libvirt.py",
> line 1178, in migrateToURI2
> if ret == -1: raise libvirtError
> ('virDomainMigrateToURI2() failed', dom=self)
> libvirtError: Failed to inquire lock: No
> such process
> Thread-26::ERROR::2014-01-17
>
> 17:01:28,917::sampling::355::__vm.Vm::(collect)
>
> vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Stats
> function failed: <AdvancedStatsFunction
> _highWrite at 0x26efb58>
>
> The problem is that it doesn't say which
> process...
>
>
> this looks like noise post the migration
> failing, with probably
> a more relevant error in libvirt log
>
>
> ----- Original Message -----
>
> From: "Itamar Heim" <iheim at redhat.com
> <mailto:iheim at redhat.com>
> <mailto:iheim at redhat.com
> <mailto:iheim at redhat.com>>>
> To: "Edgars M."
> <edgars.mazurs at gmail.com <mailto:edgars.mazurs at gmail.com>
> <mailto:edgars.mazurs at gmail.com
> <mailto:edgars.mazurs at gmail.com>>>, "Meital Bourvine"
> <mbourvin at redhat.com
> <mailto:mbourvin at redhat.com>
> <mailto:mbourvin at redhat.com <mailto:mbourvin at redhat.com>>>
> Cc: users at ovirt.org
> <mailto:users at ovirt.org> <mailto:users at ovirt.org
> <mailto:users at ovirt.org>>, "Michal
> Skrivanek" <mskrivan at redhat.com
> <mailto:mskrivan at redhat.com>
> <mailto:mskrivan at redhat.com
> <mailto:mskrivan at redhat.com>>>
> Sent: Friday, January 17, 2014 9:47:11 PM
> Subject: Re: [Users] VM Migration failed
>
> On 01/17/2014 06:25 PM, Edgars M. wrote:
>
> Hi Meital
>
> I tried to migrate another VM and
> it also failed.
>
> This is what I get in UI:
>
>
> Migration started (VM: nophpapp01,
> Source:
> novmnode1, Destination:
> novmnode2, User: edgarsm).
> Migration failed due to Error:
> Fatal error during
> migration. Trying to
> migrate to another Host (VM:
> nophpapp01, Source:
> novmnode1, Destination:
> novmnode2).
> Migration failed due to Error:
> Fatal error during
> migration (VM:
> nophpapp01, Source: novmnode1,
> Destination: novmnode2).
>
> There is nothing in
> /var/log/messages, neither on
> engine server nor nodes.
>
>
> See attachments for engine and
> vdsm logs. (I am
> migrating from vmnode1
> to vmnode2)
>
>
> whatever the issue is here, can you
> please open a bug on
> trying to
> return more info to user on the
> migration error itself
> (for easier
> troubleshooting).
>
> thanks,
> Itamar
>
>
> Thanks for the help
> Edgars
>
>
>
>
>
>
>
> On Fri, Jan 17, 2014 at 4:46 PM,
> Meital Bourvine
> <mbourvin at redhat.com
> <mailto:mbourvin at redhat.com>
> <mailto:mbourvin at redhat.com <mailto:mbourvin at redhat.com>>
> <mailto:mbourvin at redhat.com
> <mailto:mbourvin at redhat.com>
> <mailto:mbourvin at redhat.com
> <mailto:mbourvin at redhat.com>>>> wrote:
>
> Which error are you getting
> in the UI?
> Can you please attach the
> full engine and vdsm
> logs?
> Also, please check if there
> is a relevant
> error in /var/log/messages
>
>
>
> ------------------------------__------------------------------__------------
>
>
> *From: *"Edgars M."
> <edgars.mazurs at gmail.com
> <mailto:edgars.mazurs at gmail.com>
> <mailto:edgars.mazurs at gmail.com
> <mailto:edgars.mazurs at gmail.com>>
>
> <mailto:edgars.mazurs at gmail.
> <mailto:edgars.mazurs at gmail.>__com
> <mailto:edgars.mazurs at gmail.com
> <mailto:edgars.mazurs at gmail.com>>>>
> *To: *users at ovirt.org
> <mailto:users at ovirt.org>
> <mailto:users at ovirt.org
> <mailto:users at ovirt.org>> <mailto:users at ovirt.org
> <mailto:users at ovirt.org>
> <mailto:users at ovirt.org
> <mailto:users at ovirt.org>>>
> *Sent: *Friday, January
> 17, 2014 3:42:37 PM
> *Subject: *[Users] VM
> Migration failed
>
>
> Hi
>
> I am experiencing issues
> with manual VM
> migration. VM fails to
> migrate to other node in
> the same Cluster.
> Here are some
> relevant engine.log entries:
>
> ERROR
>
> [org.ovirt.engine.core.__vdsbroker.__VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler___Worker-73) Rerun vm
> a31cfd62-26fc-4396-8a83-__1aed68c7fd39.
> Called from vds novmnode1
>
> ERROR
>
> [org.ovirt.engine.core.__vdsbroker.vdsbroker.__MigrateStatusVDSCommand]
> (pool-6-thread-49) Failed in
> MigrateStatusVDS method
>
> ERROR
>
> [org.ovirt.engine.core.__vdsbroker.vdsbroker.__MigrateStatusVDSCommand]
> (pool-6-thread-49) Error
> code migrateErr
> and error message
> VDSGenericException:
> VDSErrorException:
> Failed to
> MigrateStatusVDS, error
> = Fatal error
> during migration
>
> ERROR
>
> [org.ovirt.engine.core.__vdsbroker.vdsbroker.__MigrateStatusVDSCommand]
> (pool-6-thread-49) Command
> MigrateStatusVDS execution failed.
> Exception:
> VDSErrorException:
> VDSGenericException:
> VDSErrorException: Failed to
> MigrateStatusVDS, error = Fatal
> error during migration
>
> Both Engine and Node are
> running CentOS
> 6.5 x64. oVirt Engine
> Version: 3.3.2-1.el6.
> VDSM version
> 4.13.2-1. I restarted engine
> and vdsm, that did not help.
>
> Also in vdsm log file I
> see the following
> errors related to the
> same VM ID:
>
> Thread-27::ERROR::2014-01-17
>
> 16:37:06,271::sampling::355::__vm.Vm::(collect)
>
> vmId=`a31cfd62-26fc-4396-8a83-__1aed68c7fd39`::Stats
> function
> failed:
> <AdvancedStatsFunction _highWrite
> at 0x26efb58>
>
> Any hints?
>
> BR
> Edgars
>
>
> _________________________________________________
> Users mailing list
> Users at ovirt.org <mailto:Users at ovirt.org>
> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
> <mailto:Users at ovirt.org
> <mailto:Users at ovirt.org> <mailto:Users at ovirt.org
> <mailto:Users at ovirt.org>>>
> http://lists.ovirt.org/__mailman/listinfo/users
>
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
>
>
>
>
> _________________________________________________
> Users mailing list
> Users at ovirt.org <mailto:Users at ovirt.org>
> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
> http://lists.ovirt.org/__mailman/listinfo/users
>
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
>
>
>
> _________________________________________________
> Users mailing list
> Users at ovirt.org <mailto:Users at ovirt.org>
> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
> http://lists.ovirt.org/__mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
>
> --
> Dafna Ron
>
>
>
>
> --
> Dafna Ron
>
>
--
Dafna Ron
More information about the Users
mailing list