[Qemu-devel] converging around a single guest agent

[Michael Roth]

On 11/16/2011 11:53 AM, Barak Azulay wrote:
> On Wednesday 16 November 2011 17:28:16 Michael Roth wrote:
>> On 11/16/2011 06:13 AM, Barak Azulay wrote:
>>> On Wednesday 16 November 2011 10:16:57 Alexander Graf wrote:
>>>> On 16.11.2011, at 08:05, Barak Azulay<bazulay at redhat.com>   wrote:
>>>>> On Wednesday 16 November 2011 02:42:30 Alexander Graf wrote:
>>>>>> On 16.11.2011, at 00:01, Michael Roth wrote:
>>>>>>> But practically-speaking, it's unavoidable that qemu-specific
>>>>>>> management tooling will need to communicate with qemu (via
>>>>>>> QMP/libqmp/HMP/etc, or by proxy via libvirt). It's through those same
>>>>>>> channels that the qemu-ga interfaces will ultimately be exposed, so
>>>>>>> the problem of qemu-ga vs. ovirt-guest-agent isn't really any
>>>>>>> different than the problem of QMP's system_powerdown/info_balloon/etc
>>>>>>> vs. ovirt-guest-agent's
>>>>>>> Shutdown/Available_Ram/etc: it's a policy decision rather than
>>>>>>> argument for choosing one project over another.
>>>>>>
>>>>>> I don't see why we shouldn't be able to just proxy whatever
>>>>>> communication happens between the guest agent and the management tool
>>>>>> through qemu. At that point qemu could talk to the guest agent just as
>>>>>> well as the management tool and everyone's happy.
>>>>>
>>>>> I'm not sure proxying all the requests to the guset through qemu is
>>>>> desirable, other than having single point of management, most of the
>>>>> calls will be pass throgh and has no interest to qemu (MITM?).
>>>>>
>>>>> There is a big advantage on direct communication (VDSM<->   agent), that
>>>>> way features can be added to the ovirt stack without the need to add it
>>>>> to the qemu.
>>>>
>>>> If we keep the protocol well-defined, we can get that for free. Just
>>>> have every command carry its own size and a request id shich the reply
>>>> also contains and suddenly you get asynchronous proxyable
>>>> communication.
>>>
>>> Sure we can keep commands synchronized in various ways the question is do
>>> we want that, there are a few down sides for that:
>>> 1 - VDSM will have to pass through 2 proxies (libvirt&   qemu) in order to
>>> deliver a message to the guest, this byiself is not such a big
>>> disadvantage but will force us to handle much more corner-cases.
>>
>> Can't rule out the possibility of corner-cases resulting from this, but
>> the practical way to look at it is VDSM will need handle libvirt/QMP
>> protocols well. The implementation of the proxying mechanism is where
>> the extra challenge comes into play, but this should be transparent to
>> the protocols VDSM speaks.
>>
>> Implementation-wise, just to give you an idea of the work involved if we
>> took this route:
>>
>> 1) ovirt-guest-agent would need to convert request/response payloads
>> from/to QMP payloads on the guest-side, which are JSON and should,
>> theoretically, mesh well with a python-based agent.
>
>
>
> ovirt-guest-agent already uses JSON for communication with VDSM
>
>
>
>>
>> 2) You'd also need a schema, similar to qemu.git/qapi-schema-guest.json,
>> to describe the calls you're proxying. The existing infrastructure in
>> QEMU will handle all the work of marshalling/unmarshalling responses
>> back to the QMP client on the host-side.
>>
>> It's a bit of extra work, but the benefit is unifying the
>> qemu/guest-level management interface into a single place that's easy
>> for QMP/libvirt to consume.
>>
>
> The issue is not whether it's possible or not or the amount of efforts need to
> be done for that to happen, either for qemu-ga or ovirt-guest-agent this work
> needs to be done.

Right, just trying to flesh out the areas where the extra corner-cases 
argument to this approach would be relevant. On the host-side the corner 
cases are something that already needs to be handled in the context of 
QMP/libvirt in general, it's in the implementation of 1) and 2) on the 
agent side where they'd potentially arise.

>
> the question is whether all comminication should go through the monitor (hence
> double proxy) or ... only a subset of the commands that are closly related to
> hypervisor functionality and separate it from general management-system
> related actions (e.g. ovirt or any other management system that wants to
> communicate to the guest).
>
>
>
>>> 2 - looking at the qemu-ga functionality (read&   write ...) do we really
>>> want to let a big chunk of data through both qemu&   libvirt rather than
>>> directtly to the comsumer (VDSM)
>>
>> VDSM isn't the only consumer however, HMP/QMP and libvirt are consumers
>> in and of themselves.
>
>
> This is not the claim it was just an example, it may as well be any other
> management system.
>
> The question still remains, if we want to pass a big chunk of data, do we want
> it to be passed through this double proxy layer ? personally I think it does
> not belong to core hypervisor management interface.

The common use case in practice has been reading things like 
/proc/meminfo, which ties into higher-level guest statistics collection.

I agree that for large data transfers this is not the ideal interface, 
but we can opt to not use the interface for such purposes at that level 
and avoid the potential for bad performance entirely.

>
>>
>>> 3 - When events are fired from the guest agent, the delay of passing it
>>> through a double proxy will have it's latency penalty (as we have
>>> experianced in the client disconnect spice event)
>>
>> Getting them out of the guest is probably the biggest factor, delivering
>> them between processes on the host is likely a small hit in comparison.
>>
>
>  From our experiance in the last 3 years this latency (passing a single proxy
> layer, libvirt) , affected the overall functionality&  behaviour. However
> using libvirt as a monitor proxy also gave us some advantages, the question is
> what are the advantages of passing ALL guest-agent protocol through qemu ?
>

There's the existing argument of coding against a single 
transport/interface which I think is generally applicable. Beyond that, 
events are a particularly good example actually: QEMU already has a 
state machine to aid lifecycle management and handle device callbacks, 
this data is then made available at the QMP-layer.

A management interface at all levels of the stack will want to know 
things like whether the guest is in the middle of a migration, or 
paused, or in an error state, which you get via QMP.

QEMU itself would benefit from additional guest-driven events like 
reboots, copy(/paste) notifications, etc, since these events can drive 
the state machine, or trigger device callbacks. We can then propagate 
the events up the stack via the existing (or a new) notification system 
for state changes.

If we bypass QEMU in reporting these events we complicate things at the 
management level: we're augmenting events reported by QMP with racy 
out-of-band events, and taking events away from QEMU that could be 
useful internally.

>
>
>>>>> I envision the agent will have 2 separate ports to listen to, one to
>>>>> communicate to qemu and one for VDSM.
>>>>
>>>> Ugh, no, I'd much prefer a single 'bus' everyone attaches to.
>>>
>>> why?
>>>
>>> I'm thinking on situation we'll need to priorities commands arriving from
>>> qemu over "management standard commands"&   info gathering, sure there
>>> are number of mechanisms to do that but it seems to me that a separation
>>> is the best way.
>>>
>>> e.g. I think we need to priorities a quiesce command from qemu over any
>>> other info/command from VDSM.
>>
>> Do you mean prioritize in terms of order of delivery? Best way to do
>> that is a single protocol with state-tracking, otherwise we're just racing.
>>
>>>> Alex
>>>>
>>>>> Barak
>>>>>
>>>>>> Alex
>