[Kimchi-devel] [v1 1/1] Migrate virtual machines

[Aline Manera]

On 11/04/2014 07:41 AM, simonjin wrote:
>
> 于 2014年11月04日 03:29, Aline Manera 写道:
>>
>> On 11/03/2014 12:12 PM, Simon Jin wrote:
>>> Signed-off-by: Simon Jin <simonjin at linux.vnet.ibm.com>
>>> ---
>>> docs/API.md | 3 +-
>>> src/kimchi/control/vms.py | 1 +
>>> src/kimchi/exception.py | 3 +
>>> src/kimchi/i18n.py | 11 ++++
>>> src/kimchi/model/vms.py | 138 
>>> ++++++++++++++++++++++++++++++++++++++++++++++
>>> 5 files changed, 155 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/docs/API.md b/docs/API.md
>>> index 92fbbd5..28dbc6e 100644
>>> --- a/docs/API.md
>>> +++ b/docs/API.md
>>> @@ -132,7 +132,8 @@ the following general conventions:
>>> It emulates the power reset button on a machine. Note that there is a
>>> risk of data loss caused by reset without the guest OS shutdown.
>>> * connect: Prepare the connection for spice or vnc
>>> -
>>> +* migrate: Migrate a VM to remote server, only support live mode 
>>> with out block migration.
>>> +
>>
>> As the "migrate" operation will require some data like host, username 
>> and password, you need to describe them too.
>> You can see the POST request to have an idea on how to do it.
>>
> OK
>>> ### Sub-resource: Virtual Machine Screenshot
>>>
>>> **URI:** /vms/*:name*/screenshot
>>> diff --git a/src/kimchi/control/vms.py b/src/kimchi/control/vms.py
>>> index 88d8a81..64b88ad 100644
>>> --- a/src/kimchi/control/vms.py
>>> +++ b/src/kimchi/control/vms.py
>>> @@ -42,6 +42,7 @@ class VM(Resource):
>>> for ident, node in sub_nodes.items():
>>> setattr(self, ident, node(model, self.ident))
>>> self.start = self.generate_action_handler('start')
>>
>>> + self.migrate = self.generate_action_handler('migrate', 
>>> 'remoteHost', 'user', 'passwd')
>>
>> Migration will take some time to complete so it is better to use 
>> self.generate_action_handler_task() which will expect a Task instead 
>> of a new resource.
>> Also self.generate_action_handler*() accepts only one parameter which 
>> is the action name.
>>
>> Example: self.generate_action_handler_task('migrate')
>>
> OK.
>> All the other parameters will come on a dict when hits the model code.
>> So on model/vms.py you will have;
>>
>> def migrate(self, name, params):
>>
>> And "params" will have all the information you need: host, username 
>> and password.
>>
> action_args should be a list/tuple, so the code should be: 
> self.migrate = self.generate_action_handler('migrate', ['remoteHost', 
> 'user', 'passwd']),
> other example is in src/kimchi/control/storagevolumes.py: self.resize 
> = self.generate_action_handler('resize', ['size'])
>
> def generate_action_handler(self, action_name, action_args=None):
> def wrapper(*args, **kwargs):
> validate_method(('POST'), self.role_key, self.admin_methods)
> try:
> self.lookup()
> if not self.is_authorized():
> raise UnauthorizedError('KCHAPI0009E')
>
> model_args = list(self.model_args)
> if action_args is not None:
> request = parse_request()
> model_args.extend(request[key] for key in action_args)
> fn = getattr(self.model, model_fn(self, action_name))
> ident = fn(*model_args)
>

Sorry about the mistake! You are right!

>
>>> self.poweroff = self.generate_action_handler('poweroff')
>>> self.shutdown = self.generate_action_handler('shutdown')
>>> self.reset = self.generate_action_handler('reset')
>>> diff --git a/src/kimchi/exception.py b/src/kimchi/exception.py
>>> index 039152a..732b3ee 100644
>>> --- a/src/kimchi/exception.py
>>> +++ b/src/kimchi/exception.py
>>> @@ -100,3 +100,6 @@ class TimeoutExpired(KimchiException):
>>>
>>> class UnauthorizedError(KimchiException):
>>> pass
>>
>>> +
>>> +class MigrationError(KimchiException):
>>> + pass
>>
>> You don't need to create a new type of error.
>> You can use OperationFailed
>>
> OK
>>> \ No newline at end of file
>>> diff --git a/src/kimchi/i18n.py b/src/kimchi/i18n.py
>>> index 75fb076..2fd1073 100644
>>> --- a/src/kimchi/i18n.py
>>> +++ b/src/kimchi/i18n.py
>>> @@ -101,6 +101,17 @@ messages = {
>>> "KCHVM0030E": _("Unable to get access metadata of virtual machine 
>>> %(name)s. Details: %(err)s"),
>>> "KCHVM0031E": _("The guest console password must be a string."),
>>> "KCHVM0032E": _("The life time for the guest console password must 
>>> be a number."),
>>> + "KCHVM0033E": _("Can't migrate to localhost."),
>>
>>> + "KCHVM0034E": _("Can't migrate vm %(name)s on Hypervisor arch 
>>> %(srcarch)s.to a different Hypervisor arch %(destarch)s."),
> X86 --> PPC
>>> + "KCHVM0035E": _("Can't migrate from %(srchyp)s to a different 
>>> Hypervisor type %(desthyp)s."),
>>
> Qemu/kvm --> Xen
>> What is the difference between those 2 messages?
>>
>>> + "KCHVM0036E": _("Can't migrate vm %(name)s,which is already in 
>>> migrating state."),
>>> + "KCHVM0037E": _("Can't migrate vm %(name)s,which isn't in running 
>>> state."),
>>
>> Does it mean I can not migrate a powered off vm?
>>
> a powered off vm can be migrated with --offline, will add this support 
> in v2.
>
>>> + "KCHVM0038E": _("Can't migrate vm %(name)s, vm has passthrough 
>>> device %(hostdevs)s."),
>>> + "KCHVM0039E": _("Can't migrate vm %(name)s, vm already exist on 
>>> destination server %(remoteHost)s."),
>>> + "KCHVM0040E": _("Failed Migrate vm %(name)s due error: %(err)s"),
>>> + "KCHVM0041E": _("Failed resume vm %(name)s from migration recover. 
>>> Details: %(err)s"),
>>> + "KCHVM0042E": _("Failed destory vm %(name)s from migration 
>>> recover. Details: %(err)s"),
>>> + "KCHVM0043E": _("Failed disconnect %(remoteuri)s when finish 
>>> migrate vm %(name)s. Details: %(err)s"),
>>>
>>> "KCHVMHDEV0001E": _("VM %(vmid)s does not contain directly assigned 
>>> host device %(dev_name)s."),
>>> "KCHVMHDEV0002E": _("The host device %(dev_name)s is not allowed to 
>>> directly assign to VM."),
>>> diff --git a/src/kimchi/model/vms.py b/src/kimchi/model/vms.py
>>> index 1089464..1c4eef0 100644
>>> --- a/src/kimchi/model/vms.py
>>> +++ b/src/kimchi/model/vms.py
>>> @@ -44,6 +44,11 @@ from kimchi.utils import import_class, 
>>> kimchi_log, run_setfacl_set_attr
>>> from kimchi.utils import template_name_from_uri
>>> from kimchi.xmlutils.utils import xpath_get_text, xml_item_update
>>
>>> +from kimchi.model.libvirtconnection import LibvirtConnection
>>> +from kimchi.utils import add_task
>>> +from kimchi.model.templates import TemplateModel
>>> +from kimchi.model.vmhostdevs import VMHostDevsModel
>>> +from kimchi.exception import MigrationError
>>
>> The imports must be in alphabetic order.
>>
> ok
>>> DOM_STATE_MAP = {0: 'nostate',
>>> 1: 'running',
>>> @@ -507,6 +512,139 @@ class VMModel(object):
>>> except libvirt.libvirtError as e:
>>> raise OperationFailed("KCHVM0019E",
>>> {'name': name, 'err': e.get_error_message()})
>>> +
>>> + def _preCheck(self, name, destConn, remoteHost):
>>> + kimchi_log.debug('precheck migrate %s' % name)
>>> +
>>> + if remoteHost == 'localhost' or '127.0.0.1':
>>> + kimchi_log.debug('vm %s is not in running, only running vm can be 
>>> migrated' % name)
>>> + raise MigrationError("KCHVM00033E", {'name': name,
>>> + 'remoteHost': remoteHost})
>>> +
>>> +
>>> + if self.conn.get().getInfo()[0] != destConn.getInfo()[0]:
>>> + kimchi_log.debug('vm %s can not migrate to different arch server.' 
>>> % (name, destConn.getInfo()[0]))
>>> + raise MigrationError("KCHVM00034E", {'name': name,
>>> + 'srcarch': self.conn.get().getType(),
>>> + 'destarch': self.destConn.getType()})
>>> +
>>> + if self.conn.get().getType() != self.destConn.getType():
>>> + kimchi_log.debug('vm %s can not migrate to a different hypervisor' 
>>> % name)
>>> + raise MigrationError("KCHVM00035E", {'name': self.name,
>>> + 'srchyp':self.conn.get().getType(),
>>> + 'desthyp': self.destConn.getType()})
>>> +
>>> + dom = self.vm.get_vm(self.name, self.conn)
>>> +
>>> + state = DOM_STATE_MAP[dom.info()[0]]
>>> + if state == 'migrating': #TODO how to set vm as migrating status ?
>>> + kimchi_log.debug('vm is already in migrating %s' % name)
>>> + raise MigrationError("KCHVM00036E", {'name': self.name,})
>>
>>> + elif state != 'running':
>>> + kimchi_log.debug('vm %s is not in running, only running vm can be 
>>> migrated' % name)
>>> + raise MigrationError("KCHVM00037E", {'name': self.name})
>>> +
>>
>> Why can't I do cold migration?
>>
>
>>> + #check if there is any passthrough devices belong to this vm
>>> + vmhostdevs = VMHostDevsModel(conn = self.conn)
>>> + hostdevs = vmhostdevs.get_list(self.name)
>>> + if hostdevs:
>>> + raise MigrationError("KCHVM00038E", {'name' : name,
>>> + 'hostdevs': hostdevs})
>>> +
>>> + if self.get_vm(name, destConn):
>>> + kimchi_log.debug('migrate vm %s already exist on %s' % (name, 
>>> destConn.getURI()))
>>> + raise MigrationError("KCHVM00039E", {'name' : name,
>>> + 'remoteHost': remoteHost})
>>> +
>>> +
>>> + def _getRemoteConn(self, remoteHost, user):
>>> + #open destination connect to migrate
>>> + transport = 'ssh'
>>> + duri = 'qemu+%s://%s@%s/system' % (transport,
>>> + user,remoteHost)
>>> +
>>> + conn = LibvirtConnection(duri)
>>> + destConn = conn.get()
>>> + return destConn
>>> +
>>> + def _preMigrate(self, name):
>>> + #can't update VM while in Migration source/destination
>>> + kimchi_log.debug('prepare migrate %s' % name)
>>> + dom = self.get_vm(name, self.conn)
>>> + dom.migrateSetMaxDowntime(0)
>>> +
>>> +
>>> + def _do_migrate(self, name, destConn, remoteHost, cb):
>>> +
>>> + cb('starting a migration')
>>> + kimchi_log.debug('migrate %s start' % name)
>>> +
>>> + flag = 0
>>> + flag |= (libvirt.VIR_MIGRATE_LIVE |
>>> + libvirt.VIR_MIGRATE_PEER2PEER |
>>> + libvirt.VIR_MIGRATE_TUNNELLED)
>>> + #libvirt.VIR_MIGRATE_ABORT_ON_ERROR)
>>> +
>>> + try:
>>> + self.vm.migrate(destConn, None, None,flag,
>>> + None, 0)
>>> + except libvirt.libvirtError as e:
>>> + kimchi_log.error('migrate %s to %s failed' % (name,
>>> + remoteHost))
>>> + self._recover(cb)
>>> + raise MigrationError('KCHPOOL0040E', {'err': e.message,
>>> + 'name': name})
>>> + finally:
>>> + self._finish(name, destConn, cb)
>>> +
>>> + def migrate(self, name, remoteHost, user, passwd ):
>>> + destconn = self._getRemoteConn(remoteHost, user)
>>> +
>>> + self._preCheck(name, destconn, remoteHost)
>>> + self._preMigrate(name)
>>> +
>>> + task_id = add_task('/vms/%s/migrate' % name, self._do_migrate, 
>>> self.objstore,
>>> + {'name' : name, 'destconn': destconn, 'remoteHost': remoteHost})
>>> +
>>> + # Record vm migrate task for future querying
>>> + try:
>>> + with self.objstore as session:
>>> + session.store('migrate', self.name, task_id)
>>> + return task_id
>>> + except Exception as e:
>>> + raise MigrationError('KCHPOOL0037E', {'err': e.message})
>>> +
>>> + def _recover(self, name, destconn, cb):
>>> + #recover
>>> + cb('recover from a failed migration',False)
>>> + kimchi_log.debug('recover from a failed migrate %s' % name)
>>> + dom = self.get_vm(name, self.conn)
>>> + state = DOM_STATE_MAP[dom.info()[0]]
>>> + if state == 'paused':
>>
>> Does it mean the VM will be paused during migration?
> will delete this.
>
> -Simon
>> My concern is because Kimchi does not deal well with paused state.
>>
>>> + try:
>>> + dom.resume()
>>> + except libvirt.libvirtError as e:
>>> + raise OperationFailed("KCHVM0041E",
>>> + {'name': name, 'err': e.get_error_message()})
>>> + try:
>>> + destvm = self.get_vm(name, destconn)
>>> + destvm.destroy()
>>> + kimchi_log.error("destory migrate vm on destination server")
>>> + #destory and undefine
>>> + except libvirt.libvirtError as e:
>>> + raise MigrationError('KCHPOOL0042E', {'name' : name,
>>> + 'err': e.message})
>>> +
>>> + def _finish(self, name, destConnc, cb):
>>> + kimchi_log.debug('finished migrate %s' % name)
>>> + try:
>>> + destConnc.close()
>>> + except libvirt.libvirtError as e:
>>> + raise MigrationError('KCHPOOL0043E', {'name' : name,
>>> + 'remoteuri' : destConnc.getURI(),
>>> + 'err': e.message})
>>> + finally:
>>> + cb('Migrate finished', True)
>>>
>>> def poweroff(self, name):
>>> dom = self.get_vm(name, self.conn)
>>
>