[Users] Problem with live migration.

Thu Apr 19 14:17:14 UTC 2012

On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <rvaknin at redhat.com> wrote:

>  On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
>
>
>
> On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin at redhat.com> wrote:
>
>>   On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
>>
>>
>>
>> On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin at redhat.com> wrote:
>>
>>>   On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
>>>
>>>
>>>
>>> On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim at redhat.com> wrote:
>>>
>>>> On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
>>>>
>>>>>  I can't live migrate the vm's in my cluster:
>>>>>
>>>>> 012-04-17 04:06:39,728 INFO
>>>>>  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)
>>>>> Running command: MigrateVmCommand internal: false. Entities affected :
>>>>>  ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM
>>>>> 2012-04-17 04:06:39,743 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)
>>>>> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,
>>>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>>>  dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
>>>>>
>>>>> migrationMethod=ONLINE), log id: 7c7201be
>>>>> 2012-04-17 04:06:39,748 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>>> (pool-5-thread-48) VdsBroker::migrate::Entered
>>>>> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,
>>>>>  dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
>>>>>  method=online
>>>>>
>>>>> 2012-04-17 04:06:39,749 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>>> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =
>>>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,
>>>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>>>  dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
>>>>>
>>>>> migrationMethod=ONLINE), log id: 3c40c2de
>>>>> 2012-04-17 04:06:39,834 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>>> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de
>>>>> 2012-04-17 04:06:39,840 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)
>>>>> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be
>>>>> 2012-04-17 04:06:42,166 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>>> (QuartzScheduler_Worker-18) vds::refreshVmList vm id
>>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds
>>>>> rhevh1.<domain>
>>>>> ignoring it in the refresh till migration is done
>>>>> [...]
>>>>> 2012-04-17 04:07:33,852 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>>> (QuartzScheduler_Worker-34) vds::refreshVmList vm id
>>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds
>>>>> rhevh1.<domain> ignoring it in the refresh till migration is done
>>>>> 2012-04-17 04:07:37,407 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>>> (QuartzScheduler_Worker-71) Rerun vm
>>>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain>
>>>>> 2012-04-17 04:07:37,414 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>>> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =
>>>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b
>>>>> 2012-04-17 04:07:37,496 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>>> (pool-5-thread-47) Failed in MigrateStatusVDS method
>>>>> 2012-04-17 04:07:37,497 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>>> (pool-5-thread-47) Error code createErr and error message
>>>>> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,
>>>>> error = Error creating the requested virtual machine
>>>>> 2012-04-17 04:07:37,497 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>>> (pool-5-thread-47) Command
>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
>>>>> return
>>>>> value
>>>>>  Class Name:
>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
>>>>> mStatus                       Class Name:
>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
>>>>> mCode                         9
>>>>> mMessage                      Error creating the requested virtual
>>>>> machine
>>>>>
>>>>>
>>>>> 2012-04-17 04:07:37,497 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>>> (pool-5-thread-47) Vds: rhevh2.<domain>
>>>>> 2012-04-17 04:07:37,497 ERROR
>>>>> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)
>>>>> Command MigrateStatusVDS execution failed. Exception:
>>>>> VDSErrorException:
>>>>> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,
>>>>> error = Error creating the requested virtual machine
>>>>> 2012-04-17 04:07:37,497 INFO
>>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>>> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b
>>>>> 2012-04-17 04:07:37,528 WARN
>>>>>  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)
>>>>> CanDoAction of action MigrateVm failed.
>>>>>
>>>>> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
>>>>>
>>>>>
>>>>> There is no problem regarding storage or anything like that. If I shut
>>>>> down the vm I can start it on the other host without any problems
>>>>> whatsoever.
>>>>> Do I need to do something to make live migration work? Trying to dig in
>>>>> the logs on the hypervisors, but there's so much stuff being logged
>>>>> there...
>>>>>
>>>>>
>>>>>  _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>
>>>> vdsm log from the host that failed to create the VM?
>>>>
>>>
>>> The vdsm log is frighteningly long, so I put it in pastebin:
>>> http://pastebin.com/BEeqFCCb
>>> I don't really see anything worthwhile myself, but on the other hand I'm
>>> not sure what I'm looking for, so it's mostly a massive wall of text to me,
>>> but it should be somewhere in that time frame.
>>>
>>>  Here is the failure, "code 9" is not too informative, can you attach
>>> the libvirtd.log file too?
>>>
>>> Thread-49287::DEBUG::2012-04-17
>>> 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with
>>> {'status': {'message': 'Error creating the requested virtual machine',
>>> 'code': 9}}
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>>  --
>>>
>>> Thanks,
>>>
>>> Rami Vaknin, QE @ Red Hat, TLV, IL.
>>>
>>>
>>  I think this is the relevant info from the libvirt.log:
>>  2012-04-17 14:30:53.768+0000: 8506: debug :
>> virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0,
>> uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>> 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 :
>> conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>> 2012-04-17 14:30:53.771+0000: 8506: error :
>> virNetClientProgramDispatchError:170 : Domain not found: no domain with
>> matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
>>
>>  What domain is this, is it the storage domain?
>>
>>
>>  Domain in libvirt is simply a vm.
>>
>>
>>
>> _______________________________________________
>> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> --
>>
>> Thanks,
>>
>> Rami Vaknin, QE @ Red Hat, TLV, IL.
>>
>>
> Alright, so what does this mean then? The machine I'm migrating to can't
> see the vm?
> That doesn't make sense since I can manually start it on that host..
>
>
> On second look at the vdsm.log I see that disk preparation in the
> destination host failed (timed out), try to take a look at vdsm.log at the
> destination.
> In addition, for some reason getVmStats returns "Virtual machine does not
> exist" right before the migration.
>
>
> Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper)
> [10.23.30.130]::call getVmStats with
> ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper)
> return getVmStats with {'status': {'message': 'Virtual machine does not
> exist', 'code': 1}}
> Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper)
> [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt',
> 'acpiEnable': 'true',.....
> Thread-49287::DEBUG::2012-04-17
> 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare)
> vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting
> 31s for path preparation
>
> Thread-49287::DEBUG::2012-04-17
> 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare)
> vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path
> preparation
>
> 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats
> with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper)
> return getVmStats with {'status': {'message': 'Virtual machine does not
> exist', 'code': 1}}
>
>
>
> _______________________________________________
> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
>
> Thanks,
>
> Rami Vaknin, QE @ Red Hat, TLV, IL.
>
>  The vdsm log is already included earlier in the conversation;
http://pastebin.com/BEeqFCCb
I don't really know where to go from here. :/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20120419/84c8d830/attachment-0001.html>