<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
<tt>(for some reason i never recd. Adam's note tho' I am subscribed
to all the 3 lists Cc'ed here, strange !<br>
Replying off from the mail fwd.ed to me from my colleague, pls
see my responses inline below. Thanks. )<br>
<br>
</tt>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite"><br>
<br>
<div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Adam Litke</b> <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:agl@us.ibm.com">agl@us.ibm.com</a>></span><br>
Date: Thu, May 31, 2012 at 7:31 PM<br>
Subject: Re: [vdsm] RFC: Writeup on VDSM-libstoragemgmt
integration<br>
To: Deepak C Shetty <<a moz-do-not-send="true"
href="mailto:deepakcs@linux.vnet.ibm.com">deepakcs@linux.vnet.ibm.com</a>><br>
Cc: <a moz-do-not-send="true"
href="mailto:libstoragemgmt-devel@lists.sourceforge.net">libstoragemgmt-devel@lists.sourceforge.net</a>,
<a moz-do-not-send="true" href="mailto:engine-devel@ovirt.org">engine-devel@ovirt.org</a>,
VDSM Project Development <<a moz-do-not-send="true"
href="mailto:vdsm-devel@lists.fedorahosted.org">vdsm-devel@lists.fedorahosted.org</a>><br>
<br>
<br>
<div class="HOEnZb">
<div class="h5">On Wed, May 30, 2012 at 03:08:46PM +0530,
Deepak C Shetty wrote:<br>
> Hello All,<br>
><br>
> I have a draft write-up on the VDSM-libstoragemgmt
integration.<br>
> I wanted to run this thru' the mailing list(s) to help
tune and<br>
> crystallize it, before putting it on the ovirt wiki.<br>
> I have run this once thru Ayal and Tony, so have some
of their<br>
> comments incorporated.<br>
><br>
> I still have few doubts/questions, which I have posted
below with<br>
> lines ending with '?'<br>
><br>
> Comments / Suggestions are welcome & appreciated.<br>
><br>
> thanx,<br>
> deepak<br>
><br>
> [Ccing engine-devel and libstoragemgmt lists as this
stuff is<br>
> relevant to them too]<br>
><br>
>
--------------------------------------------------------------------------------------------------------------<br>
><br>
> 1) Background:<br>
><br>
> VDSM provides high level API for node virtualization
management. It<br>
> acts in response to the requests sent by oVirt Engine,
which uses<br>
> VDSM to do all node virtualization related tasks,
including but not<br>
> limited to storage management.<br>
><br>
> libstoragemgmt aims to provide vendor agnostic API for
managing<br>
> external storage array. It should help system
administrators<br>
> utilizing open source solutions have a way to
programmatically<br>
> manage their storage hardware in a vendor neutral way.
It also aims<br>
> to facilitate management automation, ease of use and
take advantage<br>
> of storage vendor supported features which improve
storage<br>
> performance and space utilization.<br>
><br>
> Home Page: <a moz-do-not-send="true"
href="http://sourceforge.net/apps/trac/libstoragemgmt/"
target="_blank">http://sourceforge.net/apps/trac/libstoragemgmt/</a><br>
><br>
> libstoragemgmt (LSM) today supports C and python
plugins for talking<br>
> to external storage array using SMI-S as well as native
interfaces<br>
> (eg: netapp plugin )<br>
> Plan is to grow the SMI-S interface as needed over time
and add more<br>
> vendor specific plugins for exploiting features not
possible via<br>
> SMI-S or have better alternatives than using SMI-S.<br>
> For eg: Many of the copy offload features require to
use vendor<br>
> specific commands, which justifies the need for a
vendor specific<br>
> plugin.<br>
><br>
><br>
> 2) Goals:<br>
><br>
> 2a) Ability to plugin external storage array into
oVirt/VDSM<br>
> virtualization stack, in a vendor neutral way.<br>
><br>
> 2b) Ability to list features/capabilities and other
statistical<br>
> info of the array<br>
><br>
> 2c) Ability to utilize the storage array offload
capabilities<br>
> from oVirt/VDSM.<br>
><br>
><br>
> 3) Details:<br>
><br>
> LSM will sit as a new repository engine in VDSM.<br>
> VDSM Repository Engine WIP @ <a moz-do-not-send="true"
href="http://gerrit.ovirt.org/#change,192" target="_blank">http://gerrit.ovirt.org/#change,192</a><br>
><br>
> Current plan is to have LSM co-exist with VDSM on the
virtualization nodes.<br>
><br>
> *Note : 'storage' used below is generic. It can be a
file/nfs-export<br>
> for NAS targets and LUN/logical-drive for SAN targets.<br>
><br>
> VDSM can use LSM and do the following...<br>
> - Provision storage<br>
> - Consume storage<br>
><br>
> 3.1) Provisioning Storage using LSM<br>
><br>
> Typically this will be done by a Storage administrator.<br>
><br>
> oVirt/VDSM should provide storage admin the<br>
> - ability to list the different storage arrays
along with their<br>
> types (NAS/SAN), capabilities, free/used space.<br>
> - ability to provision storage using any of the
array<br>
> capabilities (eg: thin provisioned lun or new NFS
export )<br>
> - ability to manage the provisioned storage (eg:
resize/delete storage)<br>
<br>
</div>
</div>
I guess vdsm will need to model a new type of object (perhaps
StorageTarget) to<br>
be used for performing the above provisioning operations. Then,
to consume the<br>
provisioned storage, we could create a StorageConnectionRef by
passing in a<br>
StorageTarget object and some additional parameters. Sound
about right?<br>
</div>
</blockquote>
<br>
<tt>Sounds right to me, but I am not an expert in VDSM object model,
Saggi/Ayal/Dan can provide<br>
more inputs here. The (proposed) storage array entity in ovirt
engine can use this vdsm object to <br>
communicate and work with the storage array in doing the
provisioning work.<br>
<br>
Going ahead with the change to new Image Repository, I was
envisioning that LSM when integrated as<br>
a new repo engine will exhibit "Storage Provisioning" as a
implicit feature/capability, only then it<br>
will be picked up by the StorageTarget, else not.<br>
</tt><br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div class="im"><br>
> Once the storage is provisioned by the storage admin,
VDSM will have<br>
> to refresh the host(s) for them to be able to see the
newly<br>
> provisioned storage.<br>
<br>
</div>
How would this refresh affect currently connected storage and
running VMs?<br>
</div>
</blockquote>
<br>
<tt>I am not too sure on this... looking for more info from the
experts here. Per ayal, getDeviceInfo<br>
should help refresh, but by 'affect' are you referring to what
happens if post refresh the device<br>
IDs and/or names of the existing storage on the host changes ?
What exactly is the concern here ?<br>
</tt><br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div class="im"><br>
> 3.1.1) Potential flows:<br>
><br>
> Mgmt -> vdsm -> lsm: create LUN + LUN Mapping /
Zoning / whatever is<br>
> needed to make LUN available to list of hosts passed by
mgmt<br>
> Mgmt -> vdsm: getDeviceList (refreshes host and gets
list of devices)<br>
> Repeat above for all relevant hosts (depending on list
passed<br>
> earlier, mostly relevant when extending an existing VG)<br>
> Mgmt -> use LUN in normal flows.<br>
><br>
><br>
> 3.1.2) How oVirt Engine will know which LSM to use ?<br>
><br>
> Normally the way this works today is that user can choose
the host<br>
> to use (default today is SPM), however there are a few
flows where<br>
> mgmt will know which host to use:<br>
> 1. extend storage domain (add LUN to existing VG) - Use
SPM and make<br>
> sure *all* hosts that need access to this SD can see the
new LUN<br>
> 2. attach new LUN to a VM which is pinned to a specific
host - use this host<br>
> 3. attach new LUN to a VM which is not pinned - use a
host from the<br>
> cluster the VM belongs to and make sure all nodes in
cluster can see<br>
> the new LUN<br>
<br>
</div>
You are still going to need to worry about locking the shared
storage resource.<br>
Will libstoragemgmt have storage clustering support baked in or
will we continue<br>
to rely on SPM? If the latter is true, most/all of these
operations would still<br>
need to be done by SPM if I understand correctly.<br>
</div>
</blockquote>
<br>
<tt>The above scenarios were noted by me on behalf of Ayal.<br>
I don't think LSM will worry abt storage clustering. We are just
using LSM to 'talk' with the<br>
storage array. I am not sure if we need locking for the above
scenarios. We are just ensuring<br>
that the newly provisioned LUN is visible to the relevant hosts,
so not sure why we might need<br>
locking?<br>
</tt><br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div class="im"><br>
> Flows for which there is no clear candidate (Maybe we can
use the<br>
> SPM host itself which is the default ?)<br>
> 1. create a new disk without attaching it to any VM<br>
> 2. create a LUN for a new storage domain<br>
<br>
</div>
Yes, SPM would seem correct to me.<br>
<div class="im"><br>
> 3.2) Consuming storage using LSM<br>
><br>
> Typically this will be done by a virtualization
administrator<br>
><br>
> oVirt/VDSM should allow virtualization admin to<br>
> - Create a new storage domain using the storage on
the array.<br>
> - Be able to specify whether VDSM should use the
storage offload<br>
> capability (default) or override it to use its own
internal logic.<br>
<br>
</div>
If vdsm can make the right decisions, I would prefer that vdsm
decides when to use<br>
hardware offload and when to use software algorithms without
administrator<br>
intervention. It's another case where oVirt can provide
value-add by<br>
simplifying the configuration and providing optimal performance.<br>
</div>
</blockquote>
<br>
<tt>Per ayal, the thought was that in scenarios we know where the
storage array implementation is <br>
not optimal, we can override and tell VDSM to use its internal
logic than offload.<br>
</tt><br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div class="im"><br>
> 4) VDSM potential changes:<br>
><br>
> 4.1) How to represent a VM disk, 1 LUN = 1 VMdisk or 1 LV
= 1 VMdisk<br>
> ? which bring another question...1 array == 1 storage
domain OR 1<br>
> LUN/nfs-export on the array == 1 storage domain ?<br>
<br>
</div>
Saggi has mentioned some ideas on this topic so I will encourage
him to explain<br>
his thoughts here.<br>
</div>
</blockquote>
<br>
<tt>Looking forward to Saggi's thoughts :)</tt><br>
<br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>
<div class="h5"><br>
><br>
> Pros & Cons of each...<br>
><br>
> 1 array == 1 storage domain<br>
> - Each new vmdisk (aka volume) will be a new
lun/file on the array.<br>
> - Easier to exploit offload capabilities, as they
are available<br>
> at the LUN/File granularity<br>
> - Will there be any issues where there will be too
many<br>
> LUNs/Files ... any maxluns limit on linux hosts that we
might hit ?<br>
> -- VDSM has been tested with 1K LUNs and it
worked fine - ayal<br>
> - Storage array limitations on the number of LUNs
can be a<br>
> downside here.<br>
> - Would it be ok to share the array for hosting
another storage<br>
> domain if need be ?<br>
> -- Provided the existing domain is not
utilising all of the<br>
> free space<br>
> -- We can create new LUNs and hand it over to
anyone needed ?<br>
> -- Changes needed in VDSM to work with raw LUNs,
today it only<br>
> has support for consuming LUNs via VG/LV.<br>
><br>
> 1 LUN/nfs-export on the array == 1 storage domain<br>
> - How to represent a new vmdisk (aka vdsm volume)
if its a LUN<br>
> provisioned using SAN target ?<br>
> -- Will it be VG/LV as is done today for block
domains ?<br>
> -- If yes, then it will be difficult to exploit
offload<br>
> capabilities, as they are at LUN level, not at LV
level.<br>
> - Each new vmdisk will be a new file on the
nfs-export, assuming<br>
> offload capability is available at the file level, so
this should<br>
> work for NAS targets ?<br>
> - Can use the storage array for hosting multiple
storage domains.<br>
> -- Provision one more LUN and use it for
another storage<br>
> domain if need be.<br>
> - VDSM already supports this today, as part of
block storage<br>
> domains for LUNs case.<br>
><br>
> Note that we will allow user to do either one of the
two options<br>
> above, depending on need.<br>
><br>
> 4.2) Storage domain metadata will also include the<br>
> features/capabilities of the storage array as reported
by LSM.<br>
> - Capabilities (taken via LSM) will be stored
in the domain<br>
> metadata during storage domain create flow.<br>
> - Need changes in oVirt engine as well ( see
'oVirt Engine<br>
> potential changes' section below )<br>
<br>
</div>
</div>
Do we want to store the exact hw capabilities or some set of
vdsm chosen feature<br>
bits that are set at create time based on the discovered hw
capabilities? The<br>
difference would be that vdsm could choose which features to
enable at create<br>
time and update those features later if needed.<br>
</div>
</blockquote>
<br>
<tt>IIUC, you are saying VDSM will only look for those capabilities,
whcih are of interest to it and<br>
store it? That should be done by way LSM returning its
capabilities as part of it being a Image Repo.<br>
I am referring to how localFSRepo (def capabilities) is shown in
the PoC Saggie posted @<br>
<a href="http://gerrit.ovirt.org/#change,192" target="_blank">http://gerrit.ovirt.org/#change,192</a></tt><br>
<br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div class="im"><br>
> 4.3) VDSM to poll LSM for array capabilities on a regular
basis ?<br>
> Per ayal:<br>
> - If we have a 'storage array' entity in oVirt Engine
(see<br>
> 'oVirt Engine potential changes' section below ) then we
can have a<br>
> 'refresh capabilities' button/verb.<br>
> - We can periodically query the storage array.<br>
> - Query LSM before running operations (sounds
redundant to me,<br>
> but if it's cheap enough it could be simplest).<br>
><br>
> Probably need a combination of 1+2 (query at very low
frequency<br>
> - 1/hour or 1/day + refresh button)<br>
<br>
</div>
This problem can be aleviated by the abstraction I suggested
above. Then, LSM<br>
can be queried only when we may want to adjust the policy
connected with a<br>
particular storage target.<br>
</div>
</blockquote>
<br>
<tt>Not clear to me, can you explain more ?<br>
LSM might need to be contacted for updating the capabilities,
because storage admins can add/remove<br>
capabilities over a period of time. Many storage arrays provide
ability to enable/disable array <br>
features on demand.<br>
</tt><br>
<blockquote
cite="mid:CAGZKiBprEgibga6A115WqiJVKWofvDz7E3C--_hHce93+f3Vbg@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div>
<div class="h5"><br>
> 5) oVirt Engine potential changes - as described by
ayal :<br>
><br>
> - We will either need a new 'storage array' entity
in engine to<br>
> keep credentials, or, in case of storage array as
storage domain,<br>
> just keep this info as part of the domain at engine
level.<br>
> - Have a 'storage array' entity in oVirt Engine to
support<br>
> 'refresh capabilities' as a button/verb.<br>
> - When user during storage provisioning, selects a
LUN exported<br>
> from a storage array (via LSM), the oVirt Engine would
know from<br>
> then onwards that this LUN is being served via LSM.<br>
> It would then be able to query the capabilities
of the LUN<br>
> and show it to the virt admin during storage
consumption flow.<br>
><br>
> 6) Potential flows:<br>
> - Create snapshot flow<br>
> -- VDSM will check the snapshot offload
capability in the<br>
> domain metadata<br>
> -- If available, and override is not
configured, it will use<br>
> LSM to offload LUN/File snapshot<br>
> -- If override is configured or capability is
not available,<br>
> it will use its internal logic to create<br>
> snapshot (qcow2).<br>
><br>
> - Copy/Clone vmdisk flow<br>
> -- VDSM will check the copy offload capability
in the domain<br>
> metadata<br>
> -- If available, and override is not
configured, it will use<br>
> LSM to offload LUN/File copy<br>
> -- If override is configured or capability is
not available,<br>
> it will use its internal logic to create<br>
> snapshot (eg: dd cmd in case of LUN).<br>
><br>
> 7) LSM potential changes:<br>
><br>
> - list features/capabilities of the array. Eg: copy
offload,<br>
> thin prov. etc.<br>
> - list containers (aka pools) (present in LSM
today)<br>
> - Ability to list different types of arrays being
managed, their<br>
> capabilities and used/free space<br>
> - Ability to create/list/delete/resize volumes (
LUN or exports,<br>
> available in LSM as of today)<br>
> - Get monitoring info with object
(LUN/snapshot/volume) as<br>
> optional parameter for specific info. eg:
container/pool free/used<br>
> space, raid type etc.<br>
><br>
> Need to make sure above info is listed in a coherent
way across<br>
> arrays (number of LUNs, raid type used? free/total per<br>
> container/pool, per LUN?. Also need I/O statistics
wherever<br>
> possible.<br>
</div>
</div>
</div>
</blockquote>
<br>
<tt>I forgot to add this in the original mail.. adding it now.<br>
<br>
8) Concerns/Issues<br>
</tt><tt><br>
- Per Tony of libstoragemgmt</tt><br>
<pre wrap=""> -- Some additional things to consider.
-- Some of the array vendors may not allow multiple points of control at
the same time. e.g. you may not be able to have 2 or more nodes running
libStorageMgmt at the same time talking to the same array. NetApp
limits what things can be done concurrently.
-- LibStorageMgmt currently just provides the bits to control external
storage arrays. The plug-in daemon and the plug-ins themselves execute
unprivileged.
- How does the change from SPM to SDM will affect the above discussions ?
</pre>
<br>
</body>
</html>