Re: [Users] Problem with live migration.

On 04/18/2012 11:30 AM, Martin Hovmöller wrote: On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin(a)redhat.com&gt; wrote: > On 04/17/2012 05:36 PM, Martin Hovmöller wrote: > > > > On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin(a)redhat.com&gt; wrote: > >> On 04/17/2012 03:45 PM, Martin Hovmöller wrote: >> >> >> >> On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim(a)redhat.com&gt; wrote: >> >>> On 04/17/2012 12:47 PM, Martin Hovmöller wrote: >>> >>>> I can't live migrate the vm's in my cluster: >>>> >>>> 012-04-17 04:06:39,728 INFO >>>> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) >>>> Running command: MigrateVmCommand internal: false. Entities affected : >>>> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM >>>> 2012-04-17 04:06:39,743 INFO >>>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) >>>> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, >>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, >>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, >>>> dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>;, >>>> >>>> migrationMethod=ONLINE), log id: 7c7201be >>>> 2012-04-17 04:06:39,748 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] >>>> (pool-5-thread-48) VdsBroker::migrate::Entered >>>> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, >>>> dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>;, >>>> method=online >>>> >>>> 2012-04-17 04:06:39,749 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] >>>> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = >>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008, >>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, >>>> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, >>>> dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>;, >>>> >>>> migrationMethod=ONLINE), log id: 3c40c2de >>>> 2012-04-17 04:06:39,834 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] >>>> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de >>>> 2012-04-17 04:06:39,840 INFO >>>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) >>>> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be >>>> 2012-04-17 04:06:42,166 INFO >>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] >>>> (QuartzScheduler_Worker-18) vds::refreshVmList vm id >>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds >>>> rhevh1.<domain> >>>> ignoring it in the refresh till migration is done >>>> [...] >>>> 2012-04-17 04:07:33,852 INFO >>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] >>>> (QuartzScheduler_Worker-34) vds::refreshVmList vm id >>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds >>>> rhevh1.<domain> ignoring it in the refresh till migration is done >>>> 2012-04-17 04:07:37,407 ERROR >>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] >>>> (QuartzScheduler_Worker-71) Rerun vm >>>> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> >>>> 2012-04-17 04:07:37,414 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] >>>> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = >>>> f33ffc14-87ba-11e1-b610-e3aeca1e8008, >>>> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b >>>> 2012-04-17 04:07:37,496 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>>> (pool-5-thread-47) Failed in MigrateStatusVDS method >>>> 2012-04-17 04:07:37,497 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>>> (pool-5-thread-47) Error code createErr and error message >>>> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, >>>> error = Error creating the requested virtual machine >>>> 2012-04-17 04:07:37,497 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>>> (pool-5-thread-47) Command >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand >>>> return >>>> value >>>> Class Name: >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >>>> mStatus Class Name: >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >>>> mCode 9 >>>> mMessage Error creating the requested virtual >>>> machine >>>> >>>> >>>> 2012-04-17 04:07:37,497 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] >>>> (pool-5-thread-47) Vds: rhevh2.<domain> >>>> 2012-04-17 04:07:37,497 ERROR >>>> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) >>>> Command MigrateStatusVDS execution failed. Exception: >>>> VDSErrorException: >>>> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, >>>> error = Error creating the requested virtual machine >>>> 2012-04-17 04:07:37,497 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] >>>> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b >>>> 2012-04-17 04:07:37,528 WARN >>>> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) >>>> CanDoAction of action MigrateVm failed. >>>> >>>> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM >>>> >>>> >>>> There is no problem regarding storage or anything like that. If I shut >>>> down the vm I can start it on the other host without any problems >>>> whatsoever. >>>> Do I need to do something to make live migration work? Trying to dig in >>>> the logs on the hypervisors, but there's so much stuff being logged >>>> there... >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> >>> vdsm log from the host that failed to create the VM? >>> >> >> The vdsm log is frighteningly long, so I put it in pastebin: >> http://pastebin.com/BEeqFCCb >> I don't really see anything worthwhile myself, but on the other hand I'm >> not sure what I'm looking for, so it's mostly a massive wall of text to me, >> but it should be somewhere in that time frame. >> >> Here is the failure, "code 9" is not too informative, can you attach >> the libvirtd.log file too? >> >> Thread-49287::DEBUG::2012-04-17 >> 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with >> {'status': {'message': 'Error creating the requested virtual machine', >> 'code': 9}} >> >> >> >> _______________________________________________ >> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >> >> >> >> -- >> >> Thanks, >> >> Rami Vaknin, QE @ Red Hat, TLV, IL. >> >> > I think this is the relevant info from the libvirt.log: > 2012-04-17 14:30:53.768+0000: 8506: debug : > virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, > uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a > 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : > conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a > 2012-04-17 14:30:53.771+0000: 8506: error : > virNetClientProgramDispatchError:170 : Domain not found: no domain with > matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a' > > What domain is this, is it the storage domain? > > > Domain in libvirt is simply a vm. > > > > _______________________________________________ > Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users > > > > -- > > Thanks, > > Rami Vaknin, QE @ Red Hat, TLV, IL. > > Alright, so what does this mean then? The machine I'm migrating to can't see the vm? That doesn't make sense since I can manually start it on that host.. On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination. In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration. Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',..... Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} _______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users -- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. The vdsm log is already included earlier in the conversation;