[VDSM][JSON RPC] new vdsClient?
Hi everyone, the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul or at least I haven't ever found a detailed plan :) Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess. At least, we can to put together a wishlist for a new vdsClient and add it to the wiki. My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line Thoughts? -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, November 14, 2014 5:34:57 PM Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
[...]
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: [...]
Hit 'send' too early: * client library package: we should have a program-friendly python package to access VDSM, in order to make it more testable, and vdsClient should be a thin layer around that * schema-based: the package/tool should be given a schema file, and determine the API from that. * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line * JSON for output as well (seems nice, no real use case in mind) -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
First let me say kudos on kickstarting the discussion having a CLI client is essential for admins and (as the current CLI can show us) it's important to get it right. We currently have a Python client used in testing. It's efficient and low level enough to be used by any consumer of the VDSM API. It can handle large number of connections to VDSMs and you can use it in sync\async\cooperative situations. There might be a place to give something a bit simpler to set up for testing and simple scripts. I have WIP patches locally, I'll send them when I get a chance. As for a CLI. There are important things to consider: 1. The client should be independent of the API. This is an admin tool that can't be targeted to a single version of VDSM. 2. Above the JSON-RPC level there is there concept of the queues. Specifically you have to know the queue you want to push you messages to and the queue where the responses will be returned at. We also need to keep in mind that VDSM is going to be split up to multiple subsystems with their own queues. This means the cli will need that to be specified. There will be a some sort of convention on how queues are named. But until that is done we can't really release a cli tool that we can support. Also keep in minds that queues are not broadcast (notifications are) which means that if there is an engine connected it might pop the response from the queue before the CLI would. We do intend to introduce a header "X-rpc-response-queue" to solve this but we don't know if we will manage to do it by 3.6. 3. Starting in 3.6 and forward a lot of operations will slowly move from request\response to a more advanced model of asynchronous messaging. This means that copyImage() will just return if the copy was started and not if it's done. Progress information and errors\etc will be sent as notifications and will be polled outside the scope of the request itself. The CLI will obviously be limited to polling. The current model I have in mind is: vdsm-cli [options] <subsystem> [<method name> [argument, ...]] If no method name specified the program will read raw json requests from stdin and push raw responses to stdout. -a --address the remote address when omitted uses the environment variable VDSM_CLI_ADDRESS. if both are omitted tries to connect to the local broker in /var/run/vsdm/broker.socket . eg. stomp://example.com stmop+ssl://foo.com -r --raw print the raw json result. -n --nowait don't wait for a response. -------------------------------------------- Which will return either the result or error fields (only one of them should be returned). You would differentiate using the return code. Unless the user uses --raw the response should be returned in a format similar to yaml: key: value list_key: a, b, c, d map_key: key2: value key3: value text_with_multiple_lines: this text has multiple lines in-did it does another_key: 231 Having the stdin\stdout option will solve most cases where users want to send multiple requests but don't want to use python and don't want to implement STOMP\JSON_RPC. (even in shellscript [1]) For developers we are working on a sugary wrapper around the low level Python API so you could use IPython to debug VSDM. I want to encourage developers to write python tests instead of testing using the CLI. It should be out in the next few days. ``` import vdsm.client client = client.ClientFactory("stomp://address:port/_local/vdsm/requests") client.doX(args) cb = client.doX_async(args) cb.wait() client.waitForEvent("event.*.mask") ``` [1] http://www.rolandfg.net/2014/07/29/json-commandline-processor/ ----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, November 14, 2014 6:44:52 PM Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, November 14, 2014 5:34:57 PM Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
[...]
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: [...]
Hit 'send' too early:
* client library package: we should have a program-friendly python package to access VDSM, in order to make it more testable, and vdsClient should be a thin layer around that * schema-based: the package/tool should be given a schema file, and determine the API from that. * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line * JSON for output as well (seems nice, no real use case in mind)
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, 14 November, 2014 6:34:57 PM Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode). +1 JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is. Nir
On Fri, Nov 14, 2014 at 5:52 PM, Nir Soffer <nsoffer@redhat.com> wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, 14 November, 2014 6:34:57 PM Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is.
I started discussion about it and here is what I was able to gather: - we should start new vdsClient like tool which holds current vdsClient command line interface. - vdsClient should stay as it is and we can extend new client with additional functionality as we need (adding debug mode etc) - there was a suggestion to reuse ovirt-engine-cli ideas and it should use schema to determine the API
Nir _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Piotr Kliczewski" <piotr.kliczewski@gmail.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Francesco Romani" <fromani@redhat.com>, devel@ovirt.org Sent: Friday, 14 November, 2014 9:44:50 PM Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
On Fri, Nov 14, 2014 at 5:52 PM, Nir Soffer <nsoffer@redhat.com> wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Friday, 14 November, 2014 6:34:57 PM Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is.
I started discussion about it and here is what I was able to gather: - we should start new vdsClient like tool which holds current vdsClient command line interface. - vdsClient should stay as it is and we can extend new client with additional functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better. - The interface is not consistent, trying to convert command line options to xmlrpc call - Error handling is not helpful, showing an error message and then printing help for all verbse, so the error message is hidden above your screen - Help message is not useful, we need only the names of the parameters - We need extended help for specific verb - like "git help commit" - Command output is not consistent and unusable for parsing in scripts We need to print the returned value from vdsm as is, or prety printed.
- there was a suggestion to reuse ovirt-engine-cli ideas and it should use schema to determine the API
Using the schema can be nice to generate help for the commands. I started a simple jsonrpc client that can be used for testing *now*: http://gerrit.ovirt.org/35181 Example usage: Calling method without arguements: # jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] } Calling method with arguements: # jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] } Nir
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
I started discussion about it and here is what I was able to gather: - we should start new vdsClient like tool which holds current vdsClient command line interface. - vdsClient should stay as it is and we can extend new client with additional functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better.
and never look back. it's beyond redemtion.
- The interface is not consistent, trying to convert command line options to xmlrpc call - Error handling is not helpful, showing an error message and then printing help for all verbse, so the error message is hidden above your screen - Help message is not useful, we need only the names of the parameters - We need extended help for specific verb - like "git help commit" - Command output is not consistent and unusable for parsing in scripts We need to print the returned value from vdsm as is, or prety printed.
- there was a suggestion to reuse ovirt-engine-cli ideas and it should use schema to determine the API
Using the schema can be nice to generate help for the commands.
I started a simple jsonrpc client that can be used for testing *now*: http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] }
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] }
Thanks, this looks very nice and ready for contrib, imho. I'd love to see something similar done to make vdscli connect over jsonrpc. When I need to do something less than trivial from the command line, I start python and from vdsm import vdscli s = vdscli.connect() # does a horribly useful heuristic, to connect # to local host over xmlrpc print s.getVMList() I'd like to see vdscli.connect() extended to jsonrpc. This would simplify Bug 1112594 - Use jsonrpc during migration of vms too. Dan.
On Wed, Nov 19, 2014 at 6:21 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
I started discussion about it and here is what I was able to gather: - we should start new vdsClient like tool which holds current vdsClient command line interface. - vdsClient should stay as it is and we can extend new client with additional functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better.
and never look back. it's beyond redemtion.
- The interface is not consistent, trying to convert command line options to xmlrpc call - Error handling is not helpful, showing an error message and then printing help for all verbse, so the error message is hidden above your screen - Help message is not useful, we need only the names of the parameters - We need extended help for specific verb - like "git help commit" - Command output is not consistent and unusable for parsing in scripts We need to print the returned value from vdsm as is, or prety printed.
- there was a suggestion to reuse ovirt-engine-cli ideas and it should use schema to determine the API
Using the schema can be nice to generate help for the commands.
I started a simple jsonrpc client that can be used for testing *now*: http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] }
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] }
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over jsonrpc. When I need to do something less than trivial from the command line, I start python and
from vdsm import vdscli s = vdscli.connect() # does a horribly useful heuristic, to connect # to local host over xmlrpc print s.getVMList()
I'd like to see vdscli.connect() extended to jsonrpc. This would simplify
We briefly talked about it with Saggi. There is already jsonrpc client in python which is used in UT. it needs to be polished a bit.
Bug 1112594 - Use jsonrpc during migration of vms too.
Dan.
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Piotr Kliczewski" <piotr.kliczewski@gmail.com>, devel@ovirt.org Sent: Wednesday, November 19, 2014 7:21:09 PM Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far: * client library package: we should have a program-friendly python package to access VDSM, in order * JSON for input: I's like to open logs, grab a line including a command and feed vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
Actually it is more useful.
I started a simple jsonrpc client that can be used for testing *now*: http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] }
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] }
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over jsonrpc. When I need to do something less than trivial from the command line, I start python and
from vdsm import vdscli s = vdscli.connect() # does a horribly useful heuristic, to connect # to local host over xmlrpc
Maybe: s = vdscli.connect("jsonrpc:10.35.0.174:54321")
print s.getVMList()
Nice - but we have mismatch between the API (used by jsonrpc) and xmlrpc verbs.
I'd like to see vdscli.connect() extended to jsonrpc. This would simplify Bug 1112594 - Use jsonrpc during migration of vms too.
Nir
participants (5)
-
Dan Kenigsberg -
Francesco Romani -
Nir Soffer -
Piotr Kliczewski -
Saggi Mizrahi