[Engine-devel] [RFC] New Connection Management API

Tue Jan 24 17:43:39 UTC 2012

On 23/01/12 23:54, Saggi Mizrahi wrote:
> I have begun work at changing how API clients can control storage connections when interacting with VDSM.
> 
> Currently there are 2 API calls:
> connectStorageServer() - Will connect to the storage target if the host is not already connected to it.
> disconnectStorageServer() - Will disconnect from the storage target if the host is connected to it.
> 
> This API is very simple but is inappropriate when multiple clients and flows try to access the same storage.
> 
> This is currently solved by trying to synchronize things inside rhevm. This is hard and convoluted. It also brings out issues with other clients using the VDSM API.
> 
> Another problem is error recovery. Currently ovirt-engine(OE) has no way of monitoring the connections on all the hosts an if a connection disappears it's OE's responsibility to reconnect.
> 
> I suggest a different concept where VDSM 'manages' the connections. VDSM receives a manage request with the connection information and from that point forward VDSM will try to keep this connection alive. If the connection fails VDSM will automatically try and recover.
> 
> Every manage request will also have a connection ID(CID). This CID will be used when the same client asks to unamange the connection.
> When multiple requests for manage are received to the same connection they all have to have their own unique CID. By internally mapping CIDs to actual connections VDSM can properly disconnect when no CID is addressing the connection. This allows each client and even each flow to have it's own CID effectively eliminating connect\disconnect races.
> 
> The change from (dis)connect to (un)manage also changes the semantics of the calls significantly.
> Whereas connectStorageServer would have returned when the storage is either connected or failed to connect, manageStorageServer will return once VDSM registered the CID. This means that the connection might not be active immediately as the VDSM tries to connect. The connection might remain down for a long time if the storage target is down or is having issues.
> 
> This allows for VDSM to receive the manage request even if the storage is having issues and recover as soon as it's operational without user intervention.
> 
> In order for the client to query the current state of the connections I propose getStorageConnectionList(). This will return a mapping of CID to connection status. The status contains the connection info (excluding credentials), whether the connection is active, whether the connection is managed (unamanged connection are returned with transient IDs), and, if the connection is down, the last error information.
> 
> The same actual connection can return multiple times, once for each CID.
> 
> For cases where an operation requires a connection to be active a user can poll the status of the CID. The user can then choose to poll for a certain amount of time or until an error appears in the error field of the status. This will give you either a timeout or a "try once" semantic depending on the flows needs.
> 
> All connections that have been managed persist VDSM restart and will be managed until a corresponding unmanage command has been issued.
> 
> There is no concept of temporary connections as "temporary" is flow dependent and VDSM can't accommodate all interpretation of "temporary". An ad-hoc mechanism can be build using the CID field. For instance a client can manage a connection with "ENGINE_FLOW101_CON1". If the flow got interrupted the client can clean all IDs with certain flow IDs.
> 
> I think this API gives safety, robustness, and implementation freedom.
> 
> 
> Nitty Gritty:
> 
> manageStorageServer
> ===================
> Synopsis:
> manageStorageServer(uri, connectionID):
> 
> Parameters:
> uri - a uri pointing to a storage target (eg: nfs://server:export, iscsi://host/iqn;portal=1)
> connectionID - string with any char except "/".
> 
> Description:
> Tells VDSM to start managing the connection. From this moment on VDSM will try and have the connection available when needed. VDSM will monitor the connection and will automatically reconnect on failure.
> Returns:
> Success code if VDSM was able to manage the connection.
> It usually just verifies that the arguments are sane and that the CID is not already in use.
> This doesn't mean the host is connected.
> ----
> unmanageStorageServer
> =====================
> Synopsis:
> unmanageStorageServer(connectionID):
> 
> Parameters:
> connectionID - string with any char except "/".
> 
> Descriptions:
> Tells VDSM to stop managing the connection. VDSM will try and disconnect for the storage target if this is the last CID referencing the storage connection.
> 
> Returns:
> Success code if VDSM was able to unmanage the connection.
> It will return an error if the CID is not registered with VDSM. Disconnect failures are not reported. Active unmanaged connections can be tracked with getStorageServerList()
> ----
> getStorageServerList
> ====================
> Synopsis:
> getStorageServerList()
> 
> Description:
> Will return list of all managed and unmanaged connections. Unmanaged connections have temporary IDs and are not guaranteed to be consistent across calls.
> 
> Results:VDSM was able to manage the connection.
> It usually just verifies that the arguments are sane and that the CID is not already in use.
> This doesn't mean the host is connected.
> ----
> unmanageStorageServer
> =====================
> Synopsis:
> unmanageStorageServer(connectionID):
> 
> Parameters:
> connectionID - string with any char except "/".
> 
> Descriptions:
> Tells VDSM to stop managing the connection. VDSM will try and disconnect for the storage target if this is the last CID referencing the storage connection.
> 
> Returns:
> Success code if VDSM was able to unmanage the connection.
> It will return an error if the CID is not registered with VDSM. Disconnect failures are not reported. Active unmanaged connections can be tracked with getStorageServerList()
> ----
> getStorageServerList
> ====================
> Synopsis:
> getStorageServerList()
> 
> Description:
> Will return list of all managed and unmanaged connections. Unmanaged connections have temporary IDs and are not guaranteed to be consistent across calls.
> 
> Results:
> A mapping between CIDs and the status.
> example return value (Actual key names may differ)
> 
> {'conA': {'connected': True, 'managed': True, 'lastError': 0, 'connectionInfo': {
>     'remotePath': 'server:/export
>     'retrans': 3
>     'version': 4
>     }}
>  'iscsi_session_34': {'connected': False, 'managed': False, 'lastError': 339, 'connectionIfno': {
>     'hostname': 'dandylopn'
>     'portal': 1}}
> }
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel

Hi Saggi,

I see the added value in the above functionality and I think it is a
needed functionality in VDSM.

Your suggestion includes 2 concepts:
- Persist connection - auto-reconnect on failures
- Reference counting (with CID granularity)

Here are some comments:

* Assuming you meant that the new API will be a replacement to the
current API (based on previous chats we had on this topic) I think you
are missing a needed functionality to support non-persisted connection.
creating a storage domain is an example where it can be useful.
The flow includes connecting the host to the storage server creating
storage domain and disconnecting from the storage server.
Let's assume VDSM hangs while creating the storage domain any
unmanageStorageServer will fail, the engine rolls back and tries to
create the storage domain on another host, there is no reason for the
host to reconnect to this storage server.
In the above flow i would use non-persist connection if I had one.

* In the suggested solution the connect will not initiate an immediate
connect to the storage server instead it will register the connection as
handled connection and will actually generate the connect as part of the
managed connection mechanism.
I argue that this modeling is implementation driven which is wrong from
the user perspective. As a user I expect connect to actually initiate a
connect action and that the return value should indicate if the connect
succeeded, the way you modeled it the API will return true if you
succeeded 'registering' the connect.
You modeled the API to be asynchronous with no handler (task id) to
monitor the results of the action, which requires polling in the create
storage domain flow which I really don't like.
In addition you introduced a verb for monitoring the status of the
connections alone I would like to be able to monitor it as part of the
general host status and not have to poll on a new verb in addition to
the current one.

As part of solving the connection management flows in OE I am missing:

- A way to clear all managed connections.
use case: We move a host from one data center to another and we want the
host to clear all the managed connections.
we can ask for the list of managed connection and clear them but having
clearAll is much easier.

- Handling a list of Ids in each API verb

- A verb which handles create storage domain and encapsulates the
connect create and disconnect.

Thanks, Livnat