[Kimchi-devel] [RFC] Improve task management for kimchi

[Wen Wang]

On 7/2/2014 11:06 PM, Sheldon wrote:
> On 06/27/2014 07:15 PM, Wen Wang wrote:
>> Dear all,
>> *
>> **Problems:*
>> Now our strategy  for long time operation is using task which the 
>> browser needs to check up-to-date task status time by time until the 
>> task ends. It's time consuming and less efficient. Also there exists 
>> several problems when locating each task when doing debug generating 
>> and storage pool as well as some new features that might use task 
>> strategy in the future.
>>
>> *Solution*:
>> As talked with Sheldon and Zhengsheng, we came up with a solution 
>> that avoid browser checking status every 200ms. Also, we might need 
>> some more labels in each task to provide more information when 
>> getting the task like we might need to indicate which operation 
>> triggered certain task. What's in our mind is to use the strategy 
>> that allow the server inform browser about the task information. Our 
>> proposal is designed as follows.
>>
>> 1) Browser needs to register to the back end to indicate which part 
>> the result needs to reply to when the task finished.
>> 2) The back end use broker to manage message distribution: when a 
>> task is finished or experiencing an error, back end inform the 
>> browser certain part of work is finished or error.
>> 3) Using websocket of cherrypy to accomplish the message transfer.
> Now let me elaborate above.
>
> For Browser, it can be an event loop worker.
> It can subscribe event message that users care to the back end broker.
> listen the events from the broker and take some action for the event.
>
> For back:
> The broker should collect and store the events from everywhere.
> The broker should dispatch the message to the client who subscribes it.
> For some event, broker should determined whether it should dispatch to 
> user.
> Such a VM shutdown event, the broker just send it to the user who has 
> the access permission. (Yu Xing's suggestion)
> We had better define the event message format.
>
> We had better to find an existing python lib for it. If no we should 
> code it for ourself.
>                                                            [ client1 ]
>                                             care VM     |       ^
> libvirt event -------------\        shutdown  |        |  VM shutdown
>                                    \                         V       |
> event 1 -------- -------------------------->[   broker ]
>                                   /                        ^        |  
> dispatch
> event 2       ------------/                          |         |
>                                           subscribe   |        V listen
>                                                           [ client2 ]
>
>
> For websocket:
> There's also an issue about it. 
> https://github.com/kimchi-project/kimchi/issues/22
> The websocket is the pipe to connect the broker of the UI event worker.
> We should support websocket proxying directly from the cherrypy server 
> on its given port.
> Zheng Sheng is working on it.  It can work on cherrypy. Seems 
> something wrong with nginx.
>
Thanks Sheldon, as we discussed yesterday on scrum meeting, it's better 
to use the socket method solve long time task problem. We wanted to make 
this feature a generic one so that every long-time task will benefit 
from it. I think Kimchi should use the user information alongside with 
task type to decide which user(s) we should send the message to. It all 
happen when automatically. I don't think we need the message management 
for users.

Also, do we need to change all the tasks to this socket pattern or just 
for long-time tasks? I recommend long-time tasks.

For the message format, I think we might need the user name, role and 
the url to indicate which kind of task we are running.

As you said , the broker filter the message from the backend and I might 
need to know your design for the message frontend gets. We need to have 
it discussed further more.
>
>
>>
>> Best Regards
>>
>> Wang Wen
>>
>>
>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
>
> -- 
> Thanks and best regards!
>
> Sheldon Feng(???)<shaohef at linux.vnet.ibm.com>
> IBM Linux Technology Center

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/kimchi-devel/attachments/20140703/e2c7c184/attachment.html>