[Users] Problem with live migration.
Rami Vaknin
rvaknin at redhat.com
Sun Apr 22 12:40:38 UTC 2012
On 04/19/2012 05:17 PM, Martin Hovmöller wrote:
>
>
> On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <rvaknin at redhat.com
> <mailto:rvaknin at redhat.com>> wrote:
>
> On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
>>
>>
>> On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin at redhat.com
>> <mailto:rvaknin at redhat.com>> wrote:
>>
>> On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
>>>
>>>
>>> On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin
>>> <rvaknin at redhat.com <mailto:rvaknin at redhat.com>> wrote:
>>>
>>> On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
>>>>
>>>>
>>>> On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim
>>>> <iheim at redhat.com <mailto:iheim at redhat.com>> wrote:
>>>>
>>>> On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
>>>>
>>>> I can't live migrate the vm's in my cluster:
>>>>
>>>> 012-04-17 04:06:39,728 INFO
>>>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>>>> (pool-5-thread-48)
>>>> Running command: MigrateVmCommand internal:
>>>> false. Entities affected :
>>>> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM
>>>> 2012-04-17 04:06:39,743 INFO
>>>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>>>> (pool-5-thread-48)
>>>> START, MigrateVDSCommand(vdsId =
>>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>> srcHost=10.23.30.130,
>>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>> dstHost=10.23.30.110:54321
>>>> <http://10.23.30.110:54321>
>>>> <http://10.23.30.110:54321>,
>>>>
>>>> migrationMethod=ONLINE), log id: 7c7201be
>>>> 2012-04-17 04:06:39,748 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>> (pool-5-thread-48) VdsBroker::migrate::Entered
>>>> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>> srcHost=10.23.30.130,
>>>> dstHost=10.23.30.110:54321
>>>> <http://10.23.30.110:54321>
>>>> <http://10.23.30.110:54321>, method=online
>>>>
>>>> 2012-04-17 04:06:39,749 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>> (pool-5-thread-48) START,
>>>> MigrateBrokerVDSCommand(vdsId =
>>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>> srcHost=10.23.30.130,
>>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>> dstHost=10.23.30.110:54321
>>>> <http://10.23.30.110:54321>
>>>> <http://10.23.30.110:54321>,
>>>>
>>>> migrationMethod=ONLINE), log id: 3c40c2de
>>>> 2012-04-17 04:06:39,834 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>> (pool-5-thread-48) FINISH,
>>>> MigrateBrokerVDSCommand, log id: 3c40c2de
>>>> 2012-04-17 04:06:39,840 INFO
>>>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>>>> (pool-5-thread-48)
>>>> FINISH, MigrateVDSCommand, return:
>>>> MigratingFrom, log id: 7c7201be
>>>> 2012-04-17 04:06:42,166 INFO
>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>> (QuartzScheduler_Worker-18) vds::refreshVmList
>>>> vm id
>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is
>>>> migrating to vds rhevh1.<domain>
>>>> ignoring it in the refresh till migration is done
>>>> [...]
>>>> 2012-04-17 04:07:33,852 INFO
>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>> (QuartzScheduler_Worker-34) vds::refreshVmList
>>>> vm id
>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status =
>>>> Paused on vds
>>>> rhevh1.<domain> ignoring it in the refresh till
>>>> migration is done
>>>> 2012-04-17 04:07:37,407 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>> (QuartzScheduler_Worker-71) Rerun vm
>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called
>>>> from vds rhevh2.<domain>
>>>> 2012-04-17 04:07:37,414 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>> (pool-5-thread-47) START,
>>>> MigrateStatusVDSCommand(vdsId =
>>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log
>>>> id: 8dfce1b
>>>> 2012-04-17 04:07:37,496 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>> (pool-5-thread-47) Failed in MigrateStatusVDS
>>>> method
>>>> 2012-04-17 04:07:37,497 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>> (pool-5-thread-47) Error code createErr and
>>>> error message
>>>> VDSGenericException: VDSErrorException: Failed
>>>> to MigrateStatusVDS,
>>>> error = Error creating the requested virtual
>>>> machine
>>>> 2012-04-17 04:07:37,497 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>> (pool-5-thread-47) Command
>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
>>>> return
>>>> value
>>>> Class Name:
>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
>>>> mStatus Class Name:
>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
>>>> mCode 9
>>>> mMessage Error creating
>>>> the requested virtual machine
>>>>
>>>>
>>>> 2012-04-17 04:07:37,497 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>> (pool-5-thread-47) Vds: rhevh2.<domain>
>>>> 2012-04-17 04:07:37,497 ERROR
>>>> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)
>>>> Command MigrateStatusVDS execution failed.
>>>> Exception: VDSErrorException:
>>>> VDSGenericException: VDSErrorException: Failed
>>>> to MigrateStatusVDS,
>>>> error = Error creating the requested virtual
>>>> machine
>>>> 2012-04-17 04:07:37,497 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>> (pool-5-thread-47) FINISH,
>>>> MigrateStatusVDSCommand, log id: 8dfce1b
>>>> 2012-04-17 04:07:37,528 WARN
>>>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>>>> (pool-5-thread-47)
>>>> CanDoAction of action MigrateVm failed.
>>>> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
>>>>
>>>>
>>>> There is no problem regarding storage or
>>>> anything like that. If I shut
>>>> down the vm I can start it on the other host
>>>> without any problems
>>>> whatsoever.
>>>> Do I need to do something to make live
>>>> migration work? Trying to dig in
>>>> the logs on the hypervisors, but there's so
>>>> much stuff being logged there...
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>> vdsm log from the host that failed to create the VM?
>>>>
>>>>
>>>> The vdsm log is frighteningly long, so I put it in
>>>> pastebin: http://pastebin.com/BEeqFCCb
>>>> I don't really see anything worthwhile myself, but on
>>>> the other hand I'm not sure what I'm looking for, so
>>>> it's mostly a massive wall of text to me, but it should
>>>> be somewhere in that time frame.
>>> Here is the failure, "code 9" is not too informative,
>>> can you attach the libvirtd.log file too?
>>>
>>> Thread-49287::DEBUG::2012-04-17
>>> 14:25:39,834::clientIF::81::vds::(wrapper) return
>>> migrationCreate with {'status': {'message': 'Error
>>> creating the requested virtual machine', 'code': 9}}
>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> --
>>>
>>> Thanks,
>>>
>>> Rami Vaknin, QE @ Red Hat, TLV, IL.
>>>
>>>
>>> I think this is the relevant info from the libvirt.log:
>>> 2012-04-17 14:30:53.768+0000: 8506: debug :
>>> virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0,
>>> uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>>> 2012-04-17 14:30:53.768+0000: 8506: debug :
>>> virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0,
>>> uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>>> 2012-04-17 14:30:53.771+0000: 8506: error :
>>> virNetClientProgramDispatchError:170 : Domain not found: no
>>> domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
>>>
>>> What domain is this, is it the storage domain?
>>
>> Domain in libvirt is simply a vm.
>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>> --
>>
>> Thanks,
>>
>> Rami Vaknin, QE @ Red Hat, TLV, IL.
>>
>>
>> Alright, so what does this mean then? The machine I'm migrating
>> to can't see the vm?
>> That doesn't make sense since I can manually start it on that host..
>
> On second look at the vdsm.log I see that disk preparation in the
> destination host failed (timed out), try to take a look at
> vdsm.log at the destination.
> In addition, for some reason getVmStats returns "Virtual machine
> does not exist" right before the migration.
>
>
> Thread-49286::DEBUG::2012-04-17
> 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
> getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
> Thread-49286::DEBUG::2012-04-17
> 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
> {'status': {'message': 'Virtual machine does not exist', 'code': 1}}
> Thread-49287::DEBUG::2012-04-17
> 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call
> migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable':
> 'true',.....
> Thread-49287::DEBUG::2012-04-17
> 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)
> vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration
> destination: waiting 31s for path preparation
>
> Thread-49287::DEBUG::2012-04-17
> 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)
> vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting
> for path preparation
>
> 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call
> getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
> Thread-49286::DEBUG::2012-04-17
> 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with
> {'status': {'message': 'Virtual machine does not exist', 'code': 1}}
>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org <mailto:Users at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
> --
>
> Thanks,
>
> Rami Vaknin, QE @ Red Hat, TLV, IL.
>
> The vdsm log is already included earlier in the conversation;
> http://pastebin.com/BEeqFCCb
> I don't really know where to go from here. :/
The exact issue that caused the prepareVolume to fail is not mentioned
in the attached log.
Can you please tell whether the attached vdsm log is from the migration
source host or from the migration destination host, and attach as well
the second missing log? The vdsm log from the host where the vm should
run after the migration should contain a failed/hanged lvm command on
the vm's disk.
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20120422/8b0b8b9a/attachment-0001.html>
More information about the Users
mailing list