[Kimchi-devel] [RFC] Live migration backend + API

Fri Oct 16 18:51:32 UTC 2015

On 16/10/2015 08:40, Daniel Henrique Barboza wrote:
>
>
> On 10/16/2015 08:15 AM, Walter Niklaus wrote:
>>
>>
>> On 15.10.2015 21:18, Daniel Henrique Barboza wrote:
>>> Hi,
>>>
>>> Live migration will be one of the new features for Kimchi 2.0 and 
>>> I've been investigating it for a week or so. This is how I'm 
>>> planning to implement it:
>>>
>>> - API:
>>>
>>> The API will be similar to the API of the 'cloneGuest', but with 
>>> extra parameters:
>>>
>>> url : 'plugins/kimchi/vms/' +'vm_name' + "/migrate"
>>> type: 'POST'
>>> parameters:
>>> * destination: the IP address of the destination host
>>> * (optional, not sure if applicable) password: the password to log 
>>> in the destination via ssh tunnel
>>>
>>> In my opinion we can start with a 'migrate' button in the options of 
>>> the VM, which will open up a new window to insert the destination IP 
>>> address. This of course thinking about the old UI. I believe we can 
>>> do whatever it is done with the 'Clone' function in the case of the 
>>> new UI, with this extra window.
>>>
>>> As the migration might take a while, we'll need a progress bar or 
>>> something to demonstrate that the progress is still ongoing. Again, 
>>> the same thing as clone does :)
>> This progress bar may be a little bit tricky because depending on the 
>> workload of the VM the amount of dirty pages could increase again at 
>> a certain point. But from a user point of view I really would like to 
>> get the reality, meaning to see regression as well in order to give 
>> me the chance to react.
>
> Note that by 'progress bar' I mean an UI indicator that will simply 
> show that the process is still running. I do not believe that the API 
> will provide us with a progress status like "27% completed".
>
>>>
>>> - backend and core functionality:
>>>
>>> This is where it gets tricky. Libvirt has *a lot* of ways to do a 
>>> live migration between hosts. My approach, at least for this first 
>>> attempt, will be focusing in a single way, implement it, stabilize 
>>> it and then extend it to the other migration types, if necessary.
>>>
>> Makes a lot of sense.
>>> Same thing goes for the preset up. The only way I managed to do a 
>>> live migration was with a NFS storage pool in both hosts, same name 
>>> and same NFS server. This shared storage is necessary because the 
>>> live migration will *not* copy the disks of the VM, just the RAM 
>>> memory. I had to share ssh keys between the hosts to allow for 
>>> password less ssh sessions.
>>>
>> From what I remember there is a validation step before starting 
>> migration where libvirt is checking if the resources of the VM are 
>> available on the remote host as well. I suspect the check is that 
>> simple to define and start the domain in listen mode on the remote 
>> host by using the domain xml from the target. And I guess this one 
>> would already fail if the artifacts specified in the domain xml are 
>> not present.
>
> At first the validations we're going to do will be simple (check if 
> the VM is already being migrated, check if it's the same 
> hypervisor/architecture at source and destination). We can add more 
> validations when it's implemented, like trying to see if the 
> destination host has the resources (pci devices, available RAM) to 
> complete the migration.
>
>> Live migration has an option to perform volume migration as well ( 
>> it's "copy-storage-all" in virsh I think), but I have never tried it 
>> out.
>
> Unfortunately this migration type was deprecated starting in RHEL 7:
>
> https://access.redhat.com/solutions/60034
>
> "Live migration with non-shared storage in RHEL6/RHEV3.* is a 
> deprecated functionality, but under certain circunstances and with 
> some limitations, it works fine. In RHEL7, this feature has been 
> disabled."
>
>
>>
>> Are you planning to support also FCP attached storage ? I guess this 
>> one could work:
>>   - if both Hosts have access to the LUN
>>   - the disk-id used in the domain definition is the same on source 
>> and target:
>>         - for example by using the multipathing id like 
>> 36005076802810d504800000000002c1d
>>         - because sda on source could point to a different disk as 
>> sda on target.
>
> Any shared storage will do. I gave the NFS example because it was the 
> setup I've tested, but
> I am sure it will work with LUN/iSCSI or other types.
>
> The idea is that the VM disk volumes must be accessed by the same path 
> in both source and destination. The way that this is implemented it's 
> up to the user.
>
>>> At first, I want to relieve this first implementation from this pre 
>>> config, which will have to be done entirely by the user. If 
>>> everything goes well, I can add a few automation in this process 
>>> inside the backend before the 2.0 launch.
>>>
>>> Kimchi internals will be implemented by using async task facilities.
>>>
>>>
>>>
>>> Any questions? Comments?
>>>
>>>
>> Are you planning to offer a "cancel live migration" API and task as 
>> well ? I'm thinking of the situation where migration will not 
>> converge due to high workload on the VM and the user may decide to 
>> stop it ?
>
> Good idea. I'll see if it's viable to be in this first attempt.
>

There is an issue open on github related to that: 
https://github.com/kimchi-project/kimchi/issues/78

It is a feature request for Task resource. We need to think about it 
carefully as a stop action may trigger a recovery action and affect any 
other feature which uses the Task resource.

>> Is this migration going to be supported only from one Kimchi managed 
>> host to another or could the target host be also a non Kimchi managed 
>> host ?
>
> In theory, if you have libvirt on both sides it would work, even 
> without kimchi. The only reason I'm thinking about Kimchi to Kimchi at 
> first is because of the facilities that the Kimchi Federation can 
> provide to exchange user credentials.
>
>> Are we planning to transfer also Kimchi-template data to the target 
>> host in case this one is managed by Kimchi ?
>
> That sounds interesting. If Kimchi has this support in the Federation 
> already, why not?
>

IMO It is another feature and not necessarily related to live migration 
so we should discuss it separately.

>
> Also, let me clarify that when I say 'first attempt' I am not talking 
> about the final backend for the 2.0 release. I am hoping that I will 
> be able to release at least 2 versions of this backend before the 
> release. Ideas like cancelling an ongoing migration sounds useful and 
> I hope we can get it done before 2.0.
>
>
>
> Daniel
>
>>>
>>> Daniel
>>>
>>>
>>> _______________________________________________
>>> Kimchi-devel mailing list
>>> Kimchi-devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>