10:34 a.m.
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me that
since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to bootstrap
a new vdsClient, and incrementally enhance it as needs arises; the old vdsClient will
not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to the wiki.
My one, so far:
* client library package: we should have a program-friendly python package to access
VDSM, in order
* JSON for input: I's like to open logs, grab a line including a command and feed
vdsClient with the very same line
Thoughts?
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
10:44 a.m.
----- Original Message -----
From: "Francesco Romani" <fromani(a)redhat.com>
To: devel(a)ovirt.org
Sent: Friday, November 14, 2014 5:34:57 PM
Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
[...]
At least, we can to put together a wishlist for a new vdsClient and
add it to
the wiki.
My one, so far:
[...]
Hit 'send' too early:
* client library package: we should have a program-friendly python package to access
VDSM, in order to make it more testable, and vdsClient should be a thin layer around
that
* schema-based: the package/tool should be given a schema file, and determine the API
from that.
* JSON for input: I's like to open logs, grab a line including a command and feed
vdsClient with the very same line
* JSON for output as well (seems nice, no real use case in mind)
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
10:21 a.m.
First let me say kudos on kickstarting the discussion
having a CLI client is essential for admins and (as the
current CLI can show us) it's important to get it right.
We currently have a Python client used in testing.
It's efficient and low level enough to be used by
any consumer of the VDSM API. It can handle large
number of connections to VDSMs and you can use it
in sync\async\cooperative situations.
There might be a place to give something a bit simpler
to set up for testing and simple scripts. I have WIP
patches locally, I'll send them when I get a chance.
As for a CLI.
There are important things to consider:
1. The client should be independent of the API.
This is an admin tool that can't be targeted to a
single version of VDSM.
2. Above the JSON-RPC level there is there concept of the
queues. Specifically you have to know the queue you
want to push you messages to and the queue where the responses
will be returned at.
We also need to keep in mind that VDSM is going to be split up
to multiple subsystems with their own queues. This means the
cli will need that to be specified.
There will be a some sort of convention on how queues are named.
But until that is done we can't really release a cli tool that we
can support.
Also keep in minds that queues are not broadcast (notifications are)
which means that if there is an engine connected it might pop the
response from the queue before the CLI would.
We do intend to introduce a header "X-rpc-response-queue" to solve this
but we don't know if we will manage to do it by 3.6.
3. Starting in 3.6 and forward a lot of operations will slowly move
from request\response to a more advanced model of asynchronous
messaging. This means that copyImage() will just return if the copy
was started and not if it's done. Progress information and
errors\etc will be sent as notifications and will be polled outside
the scope of the request itself. The CLI will obviously be limited to
polling.
The current model I have in mind is:
vdsm-cli [options] <subsystem> [<method name> [argument, ...]]
If no method name specified the program will read raw json requests
from stdin and push raw responses to stdout.
-a --address the remote address when omitted uses the
environment variable VDSM_CLI_ADDRESS. if
both are omitted tries to connect to the
local broker in /var/run/vsdm/broker.socket .
eg. stomp://example.com stmop+ssl://foo.com
-r --raw print the raw json result.
-n --nowait don't wait for a response.
--------------------------------------------
Which will return either the result or error fields (only one of them
should be returned). You would differentiate using the return code.
Unless the user uses --raw the response should be returned in
a format similar to yaml:
key: value
list_key: a, b, c, d
map_key:
key2: value
key3: value
text_with_multiple_lines:
this text has multiple lines
in-did it does
another_key: 231
Having the stdin\stdout option will solve most cases where
users want to send multiple requests but don't want to use
python and don't want to implement STOMP\JSON_RPC.
(even in shellscript [1])
For developers we are working on a sugary wrapper around the low level
Python API so you could use IPython to debug VSDM. I want to encourage
developers to write python tests instead of testing using the CLI.
It should be out in the next few days.
```
import vdsm.client
client = client.ClientFactory("stomp://address:port/_local/vdsm/requests")
client.doX(args)
cb = client.doX_async(args)
cb.wait()
client.waitForEvent("event.*.mask")
```
[1] http://www.rolandfg.net/2014/07/29/json-commandline-processor/
----- Original Message -----
From: "Francesco Romani" <fromani(a)redhat.com>
To: devel(a)ovirt.org
Sent: Friday, November 14, 2014 6:44:52 PM
Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
----- Original Message -----
> From: "Francesco Romani" <fromani(a)redhat.com>
> To: devel(a)ovirt.org
> Sent: Friday, November 14, 2014 5:34:57 PM
> Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
[...]
> At least, we can to put together a wishlist for a new vdsClient and add it
> to
> the wiki.
>
> My one, so far:
[...]
Hit 'send' too early:
* client library package: we should have a program-friendly python package to
access
VDSM, in order to make it more testable, and vdsClient should be a thin
layer around
that
* schema-based: the package/tool should be given a schema file, and determine
the API
from that.
* JSON for input: I's like to open logs, grab a line including a command and
feed
vdsClient with the very same line
* JSON for output as well (seems nice, no real use case in mind)
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:52 a.m.
----- Original Message -----
From: "Francesco Romani" <fromani(a)redhat.com>
To: devel(a)ovirt.org
Sent: Friday, 14 November, 2014 6:34:57 PM
Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
Hi everyone,
the recent JSONRPC discussion and my last work around vdsClient reminded me
that
since a long time there it was vague plans for a vdsClient overhaul
or at least I haven't ever found a detailed plan :)
Since we have to add jsonrpc support anyway, it could be a good chance to
bootstrap
a new vdsClient, and incrementally enhance it as needs arises; the old
vdsClient will
not go away very soon, I guess.
At least, we can to put together a wishlist for a new vdsClient and add it to
the wiki.
My one, so far:
* client library package: we should have a program-friendly python package to
access
VDSM, in order
* JSON for input: I's like to open logs, grab a line including a command and
feed
vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good
idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We
need to print the json returned from vdsm as is.
Nir
1:44 p.m.
On Fri, Nov 14, 2014 at 5:52 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
----- Original Message -----
> From: "Francesco Romani" <fromani(a)redhat.com>
> To: devel(a)ovirt.org
> Sent: Friday, 14 November, 2014 6:34:57 PM
> Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
>
> Hi everyone,
>
> the recent JSONRPC discussion and my last work around vdsClient reminded me
> that
> since a long time there it was vague plans for a vdsClient overhaul
>
> or at least I haven't ever found a detailed plan :)
>
> Since we have to add jsonrpc support anyway, it could be a good chance to
> bootstrap
> a new vdsClient, and incrementally enhance it as needs arises; the old
> vdsClient will
> not go away very soon, I guess.
>
> At least, we can to put together a wishlist for a new vdsClient and add it to
> the wiki.
>
> My one, so far:
> * client library package: we should have a program-friendly python package to
> access
> VDSM, in order
> * JSON for input: I's like to open logs, grab a line including a command and
> feed
> vdsClient with the very same line
Currently we don't see commands json input in the logs, but this can be a good
idea (in debug mode).
+1
JSON for output: current vdsClient format replies in a mix of text and json. We
need to print the json returned from vdsm as is.
I started discussion about it and here is what I was able to gather:
- we should start new vdsClient like tool which holds current vdsClient
command line interface.
- vdsClient should stay as it is and we can extend new client with additional
functionality as we need (adding debug mode etc)
- there was a suggestion to reuse ovirt-engine-cli ideas and it should
use schema
to determine the API
Nir
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:24 p.m.
----- Original Message -----
From: "Piotr Kliczewski"
<piotr.kliczewski(a)gmail.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "Francesco Romani" <fromani(a)redhat.com>, devel(a)ovirt.org
Sent: Friday, 14 November, 2014 9:44:50 PM
Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
On Fri, Nov 14, 2014 at 5:52 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> ----- Original Message -----
>> From: "Francesco Romani" <fromani(a)redhat.com>
>> To: devel(a)ovirt.org
>> Sent: Friday, 14 November, 2014 6:34:57 PM
>> Subject: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
>>
>> Hi everyone,
>>
>> the recent JSONRPC discussion and my last work around vdsClient reminded
>> me
>> that
>> since a long time there it was vague plans for a vdsClient overhaul
>>
>> or at least I haven't ever found a detailed plan :)
>>
>> Since we have to add jsonrpc support anyway, it could be a good chance to
>> bootstrap
>> a new vdsClient, and incrementally enhance it as needs arises; the old
>> vdsClient will
>> not go away very soon, I guess.
>>
>> At least, we can to put together a wishlist for a new vdsClient and add it
>> to
>> the wiki.
>>
>> My one, so far:
>> * client library package: we should have a program-friendly python package
>> to
>> access
>> VDSM, in order
>> * JSON for input: I's like to open logs, grab a line including a command
>> and
>> feed
>> vdsClient with the very same line
>
> Currently we don't see commands json input in the logs, but this can be a
> good
> idea (in debug mode).
>
> +1
>
> JSON for output: current vdsClient format replies in a mix of text and
> json. We
> need to print the json returned from vdsm as is.
>
I started discussion about it and here is what I was able to gather:
- we should start new vdsClient like tool which holds current vdsClient
command line interface.
- vdsClient should stay as it is and we can extend new client with additional
functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better.
- The interface is not consistent, trying to convert command line options to
xmlrpc call
- Error handling is not helpful, showing an error message and then printing
help for all verbse, so the error message is hidden above your screen
- Help message is not useful, we need only the names of the parameters
- We need extended help for specific verb - like "git help commit"
- Command output is not consistent and unusable for parsing in scripts
We need to print the returned value from vdsm as is, or prety printed.
- there was a suggestion to reuse ovirt-engine-cli ideas and it
should
use schema
to determine the API
Using the schema can be nice to generate help for the commands.
I started a simple jsonrpc client that can be used for testing *now*:
http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList
{
"jsonrpc": "2.0",
"id": "0e043d83-294a-4d31-b1b6-6dc2f2747494",
"result": [
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"
]
}
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID":
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"}'
{
"jsonrpc": "2.0",
"id": "cefd25a3-6250-4123-8a56-d7047899e19e",
"result": [
{
"status": "Down",
"exitMessage": "Admin shut down from the engine",
"vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5",
"exitReason": 6,
"timeOffset": "0",
"exitCode": 0
}
]
}
Nir
11:21 a.m.
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
> >>
> >> Hi everyone,
> >>
> >> the recent JSONRPC discussion and my last work around vdsClient reminded
> >> me
> >> that
> >> since a long time there it was vague plans for a vdsClient overhaul
> >>
> >> or at least I haven't ever found a detailed plan :)
> >>
> >> Since we have to add jsonrpc support anyway, it could be a good chance to
> >> bootstrap
> >> a new vdsClient, and incrementally enhance it as needs arises; the old
> >> vdsClient will
> >> not go away very soon, I guess.
> >>
> >> At least, we can to put together a wishlist for a new vdsClient and add it
> >> to
> >> the wiki.
> >>
> >> My one, so far:
> >> * client library package: we should have a program-friendly python package
> >> to
> >> access
> >> VDSM, in order
> >> * JSON for input: I's like to open logs, grab a line including a
command
> >> and
> >> feed
> >> vdsClient with the very same line
> >
> > Currently we don't see commands json input in the logs, but this can be a
> > good
> > idea (in debug mode).
> >
> > +1
> >
> > JSON for output: current vdsClient format replies in a mix of text and
> > json. We
> > need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
> >
>
> I started discussion about it and here is what I was able to gather:
> - we should start new vdsClient like tool which holds current vdsClient
> command line interface.
> - vdsClient should stay as it is and we can extend new client with additional
> functionality as we need (adding debug mode etc)
Current vdsClient horrible - we need to replace it with something better.
and never look back. it's beyond redemtion.
- The interface is not consistent, trying to convert command line options to
xmlrpc call
- Error handling is not helpful, showing an error message and then printing
help for all verbse, so the error message is hidden above your screen
- Help message is not useful, we need only the names of the parameters
- We need extended help for specific verb - like "git help commit"
- Command output is not consistent and unusable for parsing in scripts
We need to print the returned value from vdsm as is, or prety printed.
> - there was a suggestion to reuse ovirt-engine-cli ideas and it should
> use schema
> to determine the API
Using the schema can be nice to generate help for the commands.
I started a simple jsonrpc client that can be used for testing *now*:
http://gerrit.ovirt.org/35181
Example usage:
Calling method without arguements:
# jsonrpc Host.getVMList
{
"jsonrpc": "2.0",
"id": "0e043d83-294a-4d31-b1b6-6dc2f2747494",
"result": [
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"
]
}
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID":
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"}'
{
"jsonrpc": "2.0",
"id": "cefd25a3-6250-4123-8a56-d7047899e19e",
"result": [
{
"status": "Down",
"exitMessage": "Admin shut down from the engine",
"vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5",
"exitReason": 6,
"timeOffset": "0",
"exitCode": 0
}
]
}
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over
jsonrpc. When I need to do something less than trivial from the command
line, I start python and
from vdsm import vdscli
s = vdscli.connect() # does a horribly useful heuristic, to connect
# to local host over xmlrpc
print s.getVMList()
I'd like to see vdscli.connect() extended to jsonrpc. This would
simplify
Bug 1112594 - Use jsonrpc during migration of vms
too.
Dan.
4:02 p.m.
On Wed, Nov 19, 2014 at 6:21 PM, Dan Kenigsberg <danken(a)redhat.com> wrote:
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
> > >>
> > >> Hi everyone,
> > >>
> > >> the recent JSONRPC discussion and my last work around vdsClient
reminded
> > >> me
> > >> that
> > >> since a long time there it was vague plans for a vdsClient overhaul
> > >>
> > >> or at least I haven't ever found a detailed plan :)
> > >>
> > >> Since we have to add jsonrpc support anyway, it could be a good chance
to
> > >> bootstrap
> > >> a new vdsClient, and incrementally enhance it as needs arises; the old
> > >> vdsClient will
> > >> not go away very soon, I guess.
> > >>
> > >> At least, we can to put together a wishlist for a new vdsClient and add
it
> > >> to
> > >> the wiki.
> > >>
> > >> My one, so far:
> > >> * client library package: we should have a program-friendly python
package
> > >> to
> > >> access
> > >> VDSM, in order
> > >> * JSON for input: I's like to open logs, grab a line including a
command
> > >> and
> > >> feed
> > >> vdsClient with the very same line
> > >
> > > Currently we don't see commands json input in the logs, but this can be
a
> > > good
> > > idea (in debug mode).
> > >
> > > +1
> > >
> > > JSON for output: current vdsClient format replies in a mix of text and
> > > json. We
> > > need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
> > >
> >
> > I started discussion about it and here is what I was able to gather:
> > - we should start new vdsClient like tool which holds current vdsClient
> > command line interface.
> > - vdsClient should stay as it is and we can extend new client with additional
> > functionality as we need (adding debug mode etc)
>
> Current vdsClient horrible - we need to replace it with something better.
and never look back. it's beyond redemtion.
>
> - The interface is not consistent, trying to convert command line options to
> xmlrpc call
> - Error handling is not helpful, showing an error message and then printing
> help for all verbse, so the error message is hidden above your screen
> - Help message is not useful, we need only the names of the parameters
> - We need extended help for specific verb - like "git help commit"
> - Command output is not consistent and unusable for parsing in scripts
> We need to print the returned value from vdsm as is, or prety printed.
>
> > - there was a suggestion to reuse ovirt-engine-cli ideas and it should
> > use schema
> > to determine the API
>
> Using the schema can be nice to generate help for the commands.
>
> I started a simple jsonrpc client that can be used for testing *now*:
> http://gerrit.ovirt.org/35181
>
> Example usage:
>
> Calling method without arguements:
>
> # jsonrpc Host.getVMList
> {
> "jsonrpc": "2.0",
> "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494",
> "result": [
> "b3f6fa00-b315-4ad4-8108-f73da817b5c5"
> ]
> }
>
> Calling method with arguements:
>
> # jsonrpc VM.getStats '{"vmID":
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"}'
> {
> "jsonrpc": "2.0",
> "id": "cefd25a3-6250-4123-8a56-d7047899e19e",
> "result": [
> {
> "status": "Down",
> "exitMessage": "Admin shut down from the engine",
> "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5",
> "exitReason": 6,
> "timeOffset": "0",
> "exitCode": 0
> }
> ]
> }
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over
jsonrpc. When I need to do something less than trivial from the command
line, I start python and
from vdsm import vdscli
s = vdscli.connect() # does a horribly useful heuristic, to connect
# to local host over xmlrpc
print s.getVMList()
I'd like to see vdscli.connect() extended to jsonrpc. This would
simplify
We briefly talked about it with Saggi. There is already jsonrpc client in python
which is used in UT. it needs to be polished a bit.
Bug 1112594 - Use jsonrpc during migration of vms
too.
Dan.
6:36 p.m.
----- Original Message -----
From: "Dan Kenigsberg" <danken(a)redhat.com>
To: "Nir Soffer" <nsoffer(a)redhat.com>
Cc: "Piotr Kliczewski" <piotr.kliczewski(a)gmail.com>, devel(a)ovirt.org
Sent: Wednesday, November 19, 2014 7:21:09 PM
Subject: Re: [ovirt-devel] [VDSM][JSON RPC] new vdsClient?
On Fri, Nov 14, 2014 at 04:24:17PM -0500, Nir Soffer wrote:
> > >>
> > >> Hi everyone,
> > >>
> > >> the recent JSONRPC discussion and my last work around vdsClient
> > >> reminded
> > >> me
> > >> that
> > >> since a long time there it was vague plans for a vdsClient overhaul
> > >>
> > >> or at least I haven't ever found a detailed plan :)
> > >>
> > >> Since we have to add jsonrpc support anyway, it could be a good
chance
> > >> to
> > >> bootstrap
> > >> a new vdsClient, and incrementally enhance it as needs arises; the
old
> > >> vdsClient will
> > >> not go away very soon, I guess.
> > >>
> > >> At least, we can to put together a wishlist for a new vdsClient and
> > >> add it
> > >> to
> > >> the wiki.
> > >>
> > >> My one, so far:
> > >> * client library package: we should have a program-friendly python
> > >> package
> > >> to
> > >> access
> > >> VDSM, in order
> > >> * JSON for input: I's like to open logs, grab a line including a
> > >> command
> > >> and
> > >> feed
> > >> vdsClient with the very same line
> > >
> > > Currently we don't see commands json input in the logs, but this can
be
> > > a
> > > good
> > > idea (in debug mode).
> > >
> > > +1
> > >
> > > JSON for output: current vdsClient format replies in a mix of text and
> > > json. We
> > > need to print the json returned from vdsm as is.
but a pretty-printed output can be useful, too.
Actually it is more useful.
> I started a simple jsonrpc client that can be used for testing
*now*:
> http://gerrit.ovirt.org/35181
>
> Example usage:
>
> Calling method without arguements:
>
> # jsonrpc Host.getVMList
> {
> "jsonrpc": "2.0",
> "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494",
> "result": [
> "b3f6fa00-b315-4ad4-8108-f73da817b5c5"
> ]
> }
>
> Calling method with arguements:
>
> # jsonrpc VM.getStats '{"vmID":
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"}'
> {
> "jsonrpc": "2.0",
> "id": "cefd25a3-6250-4123-8a56-d7047899e19e",
> "result": [
> {
> "status": "Down",
> "exitMessage": "Admin shut down from the engine",
> "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5",
> "exitReason": 6,
> "timeOffset": "0",
> "exitCode": 0
> }
> ]
> }
Thanks, this looks very nice and ready for contrib, imho.
I'd love to see something similar done to make vdscli connect over
jsonrpc. When I need to do something less than trivial from the command
line, I start python and
from vdsm import vdscli
s = vdscli.connect() # does a horribly useful heuristic, to connect
# to local host over xmlrpc
Maybe:
s = vdscli.connect("jsonrpc:10.35.0.174:54321")
print s.getVMList()
Nice - but we have mismatch between the API (used by jsonrpc) and xmlrpc verbs.
I'd like to see vdscli.connect() extended to jsonrpc. This would
simplify
Bug 1112594 - Use jsonrpc during migration of vms
too.
Nir
3465
days inactive
3474
days old
8 comments
5 participants
participants (5)
-
Dan Kenigsberg -
Francesco Romani -
Nir Soffer -
Piotr Kliczewski -
Saggi Mizrahi