[Users] Problem with live migration.

Rami Vaknin rvaknin at redhat.com
Wed Apr 18 09:06:13 UTC 2012


On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
>
>
> On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin at redhat.com 
> <mailto:rvaknin at redhat.com>> wrote:
>
>     On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
>>
>>
>>     On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin at redhat.com
>>     <mailto:rvaknin at redhat.com>> wrote:
>>
>>         On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
>>>
>>>
>>>         On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim
>>>         <iheim at redhat.com <mailto:iheim at redhat.com>> wrote:
>>>
>>>             On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
>>>
>>>                 I can't live migrate the vm's in my cluster:
>>>
>>>                 012-04-17 04:06:39,728 INFO
>>>                  [org.ovirt.engine.core.bll.MigrateVmCommand]
>>>                 (pool-5-thread-48)
>>>                 Running command: MigrateVmCommand internal: false.
>>>                 Entities affected :
>>>                  ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM
>>>                 2012-04-17 04:06:39,743 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>>>                 (pool-5-thread-48)
>>>                 START, MigrateVDSCommand(vdsId =
>>>                 f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>                 vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>                 srcHost=10.23.30.130,
>>>                 dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>                 dstHost=10.23.30.110:54321
>>>                 <http://10.23.30.110:54321>
>>>                 <http://10.23.30.110:54321>,
>>>
>>>                 migrationMethod=ONLINE), log id: 7c7201be
>>>                 2012-04-17 04:06:39,748 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>                 (pool-5-thread-48) VdsBroker::migrate::Entered
>>>                 (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>                 srcHost=10.23.30.130,
>>>                 dstHost=10.23.30.110:54321
>>>                 <http://10.23.30.110:54321>
>>>                 <http://10.23.30.110:54321>,  method=online
>>>
>>>                 2012-04-17 04:06:39,749 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>                 (pool-5-thread-48) START,
>>>                 MigrateBrokerVDSCommand(vdsId =
>>>                 f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>                 vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
>>>                 srcHost=10.23.30.130,
>>>                 dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
>>>                 dstHost=10.23.30.110:54321
>>>                 <http://10.23.30.110:54321>
>>>                 <http://10.23.30.110:54321>,
>>>
>>>                 migrationMethod=ONLINE), log id: 3c40c2de
>>>                 2012-04-17 04:06:39,834 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>>>                 (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand,
>>>                 log id: 3c40c2de
>>>                 2012-04-17 04:06:39,840 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>>>                 (pool-5-thread-48)
>>>                 FINISH, MigrateVDSCommand, return: MigratingFrom,
>>>                 log id: 7c7201be
>>>                 2012-04-17 04:06:42,166 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>                 (QuartzScheduler_Worker-18) vds::refreshVmList vm id
>>>                 ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to
>>>                 vds rhevh1.<domain>
>>>                 ignoring it in the refresh till migration is done
>>>                 [...]
>>>                 2012-04-17 04:07:33,852 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>                 (QuartzScheduler_Worker-34) vds::refreshVmList vm id
>>>                 ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused
>>>                 on vds
>>>                 rhevh1.<domain> ignoring it in the refresh till
>>>                 migration is done
>>>                 2012-04-17 04:07:37,407 ERROR
>>>                 [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>                 (QuartzScheduler_Worker-71) Rerun vm
>>>                 ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from
>>>                 vds rhevh2.<domain>
>>>                 2012-04-17 04:07:37,414 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>                 (pool-5-thread-47) START,
>>>                 MigrateStatusVDSCommand(vdsId =
>>>                 f33ffc14-87ba-11e1-b610-e3aeca1e8008,
>>>                 vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id:
>>>                 8dfce1b
>>>                 2012-04-17 04:07:37,496 ERROR
>>>                 [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>                 (pool-5-thread-47) Failed in MigrateStatusVDS method
>>>                 2012-04-17 04:07:37,497 ERROR
>>>                 [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>                 (pool-5-thread-47) Error code createErr and error
>>>                 message
>>>                 VDSGenericException: VDSErrorException: Failed to
>>>                 MigrateStatusVDS,
>>>                 error = Error creating the requested virtual machine
>>>                 2012-04-17 04:07:37,497 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>                 (pool-5-thread-47) Command
>>>                 org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
>>>                 return
>>>                 value
>>>                  Class Name:
>>>                 org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
>>>                 mStatus                       Class Name:
>>>                 org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
>>>                 mCode                         9
>>>                 mMessage                      Error creating the
>>>                 requested virtual machine
>>>
>>>
>>>                 2012-04-17 04:07:37,497 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
>>>                 (pool-5-thread-47) Vds: rhevh2.<domain>
>>>                 2012-04-17 04:07:37,497 ERROR
>>>                 [org.ovirt.engine.core.vdsbroker.VDSCommandBase]
>>>                 (pool-5-thread-47)
>>>                 Command MigrateStatusVDS execution failed.
>>>                 Exception: VDSErrorException:
>>>                 VDSGenericException: VDSErrorException: Failed to
>>>                 MigrateStatusVDS,
>>>                 error = Error creating the requested virtual machine
>>>                 2012-04-17 04:07:37,497 INFO
>>>                  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>                 (pool-5-thread-47) FINISH, MigrateStatusVDSCommand,
>>>                 log id: 8dfce1b
>>>                 2012-04-17 04:07:37,528 WARN
>>>                  [org.ovirt.engine.core.bll.MigrateVmCommand]
>>>                 (pool-5-thread-47)
>>>                 CanDoAction of action MigrateVm failed.
>>>                 Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
>>>
>>>
>>>                 There is no problem regarding storage or anything
>>>                 like that. If I shut
>>>                 down the vm I can start it on the other host without
>>>                 any problems
>>>                 whatsoever.
>>>                 Do I need to do something to make live migration
>>>                 work? Trying to dig in
>>>                 the logs on the hypervisors, but there's so much
>>>                 stuff being logged there...
>>>
>>>
>>>                 _______________________________________________
>>>                 Users mailing list
>>>                 Users at ovirt.org <mailto:Users at ovirt.org>
>>>                 http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>             vdsm log from the host that failed to create the VM?
>>>
>>>
>>>         The vdsm log is frighteningly long, so I put it in pastebin:
>>>         http://pastebin.com/BEeqFCCb
>>>         I don't really see anything worthwhile myself, but on the
>>>         other hand I'm not sure what I'm looking for, so it's mostly
>>>         a massive wall of text to me, but it should be somewhere in
>>>         that time frame.
>>         Here is the failure, "code 9" is not too informative, can you
>>         attach the libvirtd.log file too?
>>
>>         Thread-49287::DEBUG::2012-04-17
>>         14:25:39,834::clientIF::81::vds::(wrapper) return
>>         migrationCreate with {'status': {'message': 'Error creating
>>         the requested virtual machine', 'code': 9}}
>>
>>>
>>>
>>>         _______________________________________________
>>>         Users mailing list
>>>         Users at ovirt.org  <mailto:Users at ovirt.org>
>>>         http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>         -- 
>>
>>         Thanks,
>>
>>         Rami Vaknin, QE @ Red Hat, TLV, IL.
>>
>>
>>     I think this is the relevant info from the libvirt.log:
>>     2012-04-17 14:30:53.768+0000: 8506: debug :
>>     virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0,
>>     uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>>     2012-04-17 14:30:53.768+0000: 8506: debug :
>>     virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0,
>>     uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
>>     2012-04-17 14:30:53.771+0000: 8506: error :
>>     virNetClientProgramDispatchError:170 : Domain not found: no
>>     domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
>>
>>     What domain is this, is it the storage domain?
>
>     Domain in libvirt is simply a vm.
>
>>
>>
>>     _______________________________________________
>>     Users mailing list
>>     Users at ovirt.org  <mailto:Users at ovirt.org>
>>     http://lists.ovirt.org/mailman/listinfo/users
>
>
>     -- 
>
>     Thanks,
>
>     Rami Vaknin, QE @ Red Hat, TLV, IL.
>
>
> Alright, so what does this mean then? The machine I'm migrating to 
> can't see the vm?
> That doesn't make sense since I can manually start it on that host..

On second look at the vdsm.log I see that disk preparation in the 
destination host failed (timed out), try to take a look at vdsm.log at 
the destination.
In addition, for some reason getVmStats returns "Virtual machine does 
not exist" right before the migration.


Thread-49286::DEBUG::2012-04-17 
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
Thread-49286::DEBUG::2012-04-17 
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with 
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}
Thread-49287::DEBUG::2012-04-17 
14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',.....
Thread-49287::DEBUG::2012-04-17 
14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: 
waiting 31s for path preparation

Thread-49287::DEBUG::2012-04-17 
14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for 
path preparation

14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
Thread-49286::DEBUG::2012-04-17 
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with 
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users


-- 

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20120418/50406519/attachment-0001.html>


More information about the Users mailing list