Questions about VDSM communication protocol
Hello, 1. I know vdsm communication adopted rpc over "stomp" and i'm wondering if it's due to use of the message broker or any other purpose. 2. according to https://www.ovirt.org/develop/release-management/features/infra/jsonrpc.html, vdsm has the final plan to completely separate msg broker. Is it still valid and under development? 3. if so, why is vdsm trying to use msg broker? because it seems enough even without msg broker. thanks,
Hi Henry, Let me answer your questions at least partially. Stomp provides a simplified protocol for the messaging which at a time greatly helped effectively adding new functionalities. We do not have any development plans to separate the broker. AFAIK, in the past there were some attempts to use Rabbit MQ but that has never left the PoC stage(please somebody correct me if I am wrong). The current implementation with internal broker is performant enough (and reliable) to support huge installations (I cannot find relevant documentation right now, hopefully others will post urls) and gives enough flexibility to implement new messaging flows. By huge I mean hundreds of hosts and thousands of VMs communicating with each other. @Piotr Kliczewski <pkliczew@redhat.com> @Martin Perina <mperina@redhat.com> Would you like to add anything more? Artur On Thu, Dec 2, 2021 at 2:31 AM Henry lol <pub.virtualization@gmail.com> wrote:
Hello,
1. I know vdsm communication adopted rpc over "stomp" and i'm wondering if it's due to use of the message broker or any other purpose.
2. according to https://www.ovirt.org/develop/release-management/features/infra/jsonrpc.html, vdsm has the final plan to completely separate msg broker. Is it still valid and under development?
3. if so, why is vdsm trying to use msg broker? because it seems enough even without msg broker.
thanks, _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EREUPD7LZIFV6Y...
-- Artur Socha Senior Software Engineer, RHV Red Hat
On Thu, Dec 2, 2021 at 9:22 AM Artur Socha <asocha@redhat.com> wrote:
Hi Henry, Let me answer your questions at least partially.
Stomp provides a simplified protocol for the messaging which at a time greatly helped effectively adding new functionalities.
We do not have any development plans to separate the broker. AFAIK, in the past there were some attempts to use Rabbit MQ but that has never left the PoC stage(please somebody correct me if I am wrong). The current implementation with internal broker is performant enough (and reliable) to support huge installations (I cannot find relevant documentation right now, hopefully others will post urls) and gives enough flexibility to implement new messaging flows. By huge I mean hundreds of hosts and thousands of VMs communicating with each other.
@Piotr Kliczewski <pkliczew@redhat.com> @Martin Perina <mperina@redhat.com> Would you like to add anything more?
In general the motivation was to make the transport layer asynchronous and allow for push notifications from vdsm. In the initial phase we were exploring a need for a broker and after some exploration and testing decided not to do it. We came to a point where transport was no longer a bottleneck. There were other parts (engine mostly) which would need to be improved to scale further. Please let me know if you any further questions
Artur
On Thu, Dec 2, 2021 at 2:31 AM Henry lol <pub.virtualization@gmail.com> wrote:
Hello,
1. I know vdsm communication adopted rpc over "stomp" and i'm wondering if it's due to use of the message broker or any other purpose.
2. according to https://www.ovirt.org/develop/release-management/features/infra/jsonrpc.html, vdsm has the final plan to completely separate msg broker. Is it still valid and under development?
3. if so, why is vdsm trying to use msg broker? because it seems enough even without msg broker.
thanks, _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EREUPD7LZIFV6Y...
-- Artur Socha Senior Software Engineer, RHV Red Hat
In my understanding, an internal broker was introduced with benefits and enough to cover the workload, right? but not clear about the need for it on oVirt because I'm not sure there are so many message flows(or message topics). I felt like it acts as just a proxy only between engine and rpc server, so it looks more desirable to me to directly communicate without message queues in the middle. (even internal broker is coupled with vdsm) could I know usecases of it in oVirt compared to raw tcp? sorry in advance if it sounds foolish.
On Fri, Dec 3, 2021 at 11:22 AM Henry lol <pub.virtualization@gmail.com> wrote:
In my understanding, an internal broker was introduced with benefits and enough to cover the workload, right?
In general it is correct. There is no broker process running but rather the code is spread between the client (engine) and server (vdsm).
but not clear about the need for it on oVirt because I'm not sure there are so many message flows(or message topics).
We created a topic per type of interactions (commands) that are being called. Good number of them are defined.
I felt like it acts as just a proxy only between engine and rpc server, so it looks more desirable to me to directly communicate without message queues in the middle. (even internal broker is coupled with vdsm)
We were concerned about topology complexity and never decided to use an external (proxy) broker. The transport communication hides details so from a logic perspective it looks like direct communication. Please remember that in the past transport was fully synchronous and one of the requirements was to abstract async so the business layer can work without changes.
could I know usecases of it in oVirt compared to raw tcp?
Interesting one is bidirectional notifications which need to be delivered to specific parts of the logic on either side. For most of use cases we could get away with raw tcp.
sorry in advance if it sounds foolish.
2021년 12월 3일 (금) 오후 7:39, Piotr Kliczewski <pkliczew@redhat.com>님이 작성:
On Fri, Dec 3, 2021 at 11:22 AM Henry lol <pub.virtualization@gmail.com> wrote:
In my understanding, an internal broker was introduced with benefits and enough to cover the workload, right?
In general it is correct. There is no broker process running but rather the code is spread between the client (engine) and server (vdsm).
but not clear about the need for it on oVirt because I'm not sure there are so many message flows(or message topics).
We created a topic per type of interactions (commands) that are being called. Good number of them are defined.
I felt like it acts as just a proxy only between engine and rpc server, so it looks more desirable to me to directly communicate without message queues in the middle. (even internal broker is coupled with vdsm)
We were concerned about topology complexity and never decided to use an external (proxy) broker. The transport communication hides details so from a logic perspective it looks like direct communication. Please remember that in the past transport was fully synchronous and one of the requirements was to abstract async so the business layer can work without changes.
yeah as you said, the business layer can focus on their own logic thanks to the transport layer. I thought that the transport layer could be also abstracted and implemented from raw tcp or websocket, and at first that using msg protocol with the broker is just an additional overhead. (correct me if i'm wrong) (i.e. json-rpc communication through tcp or websocket) but as Artur said, the main reason for using stomp here seems to easily propagate and define msg flows, right? sorry again in advance if it sounds foolish.
could I know usecases of it in oVirt compared to raw tcp?
Interesting one is bidirectional notifications which need to be delivered to specific parts of the logic on either side. For most of use cases we could get away with raw tcp.
sorry in advance if it sounds foolish.
On Mon, Dec 6, 2021 at 6:24 AM Henry lol <pub.virtualization@gmail.com> wrote:
2021년 12월 3일 (금) 오후 7:39, Piotr Kliczewski <pkliczew@redhat.com>님이 작성:
On Fri, Dec 3, 2021 at 11:22 AM Henry lol <pub.virtualization@gmail.com>
In my understanding, an internal broker was introduced with benefits
and enough to cover the workload, right?
In general it is correct. There is no broker process running but rather
wrote: the code is spread between the client (engine) and server (vdsm).
but not clear about the need for it on oVirt because I'm not sure there
are so many message flows(or message topics).
We created a topic per type of interactions (commands) that are being
called. Good number of them are defined.
I felt like it acts as just a proxy only between engine and rpc server,
(even internal broker is coupled with vdsm)
We were concerned about topology complexity and never decided to use an external (proxy) broker. The transport communication hides details so from a logic perspective it looks like direct communication. Please remember that in the past
so it looks more desirable to me to directly communicate without message queues in the middle. transport was fully synchronous and one of the requirements was to abstract async so the business layer
can work without changes.
yeah as you said, the business layer can focus on their own logic thanks to the transport layer. I thought that the transport layer could be also abstracted and implemented from raw tcp or websocket, and at first that using msg protocol with the broker is just an additional overhead. (correct me if i'm wrong) (i.e. json-rpc communication through tcp or websocket)
During the transition process we had several protocols being supported so the abstractions are there. Raw tcp is there and you can implement "just" json-rpc on top of it. In the past we were using xmlrpc so with this approach you would have unidirectional communication and no notifications. It was our starting point with the transport changes we did.
but as Artur said, the main reason for using stomp here seems to easily propagate and define msg flows, right?
There is more to it. We use stomp heartbeats <https://stomp.github.io/stomp-specification-1.2.html#Heart-beating>to determine if a host is healthy. It is used to trigger fencing flow when the host is not reachable.
sorry again in advance if it sounds foolish.
could I know usecases of it in oVirt compared to raw tcp?
Interesting one is bidirectional notifications which need to be delivered to specific parts of the logic on either side. For most of use cases we could get away with raw tcp.
sorry in advance if it sounds foolish.
Thanks for the detailed explanation!! really helpful 2021년 12월 6일 (월) 오후 4:50, Piotr Kliczewski <pkliczew@redhat.com>님이 작성:
On Mon, Dec 6, 2021 at 6:24 AM Henry lol <pub.virtualization@gmail.com> wrote:
2021년 12월 3일 (금) 오후 7:39, Piotr Kliczewski <pkliczew@redhat.com>님이 작성:
On Fri, Dec 3, 2021 at 11:22 AM Henry lol <pub.virtualization@gmail.com> wrote:
In my understanding, an internal broker was introduced with benefits and enough to cover the workload, right?
In general it is correct. There is no broker process running but rather the code is spread between the client (engine) and server (vdsm).
but not clear about the need for it on oVirt because I'm not sure there are so many message flows(or message topics).
We created a topic per type of interactions (commands) that are being called. Good number of them are defined.
I felt like it acts as just a proxy only between engine and rpc server, so it looks more desirable to me to directly communicate without message queues in the middle. (even internal broker is coupled with vdsm)
We were concerned about topology complexity and never decided to use an external (proxy) broker. The transport communication hides details so from a logic perspective it looks like direct communication. Please remember that in the past transport was fully synchronous and one of the requirements was to abstract async so the business layer can work without changes.
yeah as you said, the business layer can focus on their own logic thanks to the transport layer. I thought that the transport layer could be also abstracted and implemented from raw tcp or websocket, and at first that using msg protocol with the broker is just an additional overhead. (correct me if i'm wrong) (i.e. json-rpc communication through tcp or websocket)
During the transition process we had several protocols being supported so the abstractions are there. Raw tcp is there and you can implement "just" json-rpc on top of it. In the past we were using xmlrpc so with this approach you would have unidirectional communication and no notifications. It was our starting point with the transport changes we did.
but as Artur said, the main reason for using stomp here seems to easily propagate and define msg flows, right?
There is more to it. We use stomp heartbeats to determine if a host is healthy. It is used to trigger fencing flow when the host is not reachable.
sorry again in advance if it sounds foolish.
could I know usecases of it in oVirt compared to raw tcp?
Interesting one is bidirectional notifications which need to be delivered to specific parts of the logic on either side. For most of use cases we could get away with raw tcp.
sorry in advance if it sounds foolish.
participants (3)
-
Artur Socha -
Henry lol -
Piotr Kliczewski