----- Original Message -----

From: "Adam Litke" <agl@us.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, December 4, 2012 6:08:25 PM Subject: Re: [vdsm] RFC: New Storage API

Thanks for sharing this. It's nice to have something a little more concrete to think about. Just a few comments and questions inline to get some discussion flowing.

On Tue, Dec 04, 2012 at 04:52:40PM -0500, Saggi Mizrahi wrote:

...

I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit. I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a storage domain but a storage repository. This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID. The UUID was only used in the pool members manifest and is no longer needed.

connectStorageRepository(repoId, repoFormat, connectionParameters={}):

We should probably add an options/flags parameter for extension of all new APIs. Usually I agree but connectionParameters is already generic enough :)

...

repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster. repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2). connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.

disconnectStorageRepository(self, repoId):

I assume 'self' is a mistake here. Just want to clarify given all of the recent talk about instances vs. namespaces. Yea, it's just pasted from my code

...

In the new API there are only images, some images are mutable and some are not. mutable images are also called VirtualDisks immutable images are also called Snapshots

By mutable you mean writable right? Or does the word mutable imply more than that? It's a semantic distinction due to implementation details, in general terms, yes.

...

There are no explicit templates, you can create as many images as you want from any snapshot.

There are 4 major image operations:

createVirtualDisk(targetRepoId, size, baseSnapshotId=None, userData={}, options={}):

Is userdata a 'StringMap'? currently it's a json object. We could limit it to a string map and trust the client to parse types. We can have it be a string\blob and trust the user to serialize the data. It's pass-through object either way.

I will reopen the argument about an options dict vs a flags parameter. I oppose the dict for expansion because I think it causes APIs to devolve into a mess where lots of arbitrary and not well thought out overrides are packed into the dict over time. A flags argument (in json and python it can be an enum array) limits us to really switching flags on and off instead of passing arbitrary data. We already have strategy that we know we want to have several options. Other stuff that have been suggested is to be able to override the img format (qcow2\qed)

The way I envision it is having an class opts = CommandOptions() that you add opts.addStringOption("key", "value") opts.addIntOption("key", 3) opt.addBoolOption("key", True) I know you could just as well have strategy_space_flag and strategy_performance_flag and fail the operation if they both exist. Since it is a matter of personal taste I think it should be decided by a vote.

...

targetRepoId - ID of a connected repo where the disk will be created size - The size of the image you wish to create baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on userData - optional data that will be attached to the new VD, could be anything that the user desires. options - options to modify VDSMs default behavior

returns the id of the new VD

createSnapshot(targetRepoId, baseVirtualDiskId, userData={}, options={}): targetRepoId - The ID of a connected repo where the new sanpshot will be created and the original image exists as well. size - The size of the image you wish to create

Why is this needed? Doesn't the size of a snapshot have to be equal to its base image?

oops, another copy\paste error, you can see this arg doesn't exist in the method signature. My proof reading do need more work.

...

baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot

Can you snapshot a snapshot? In that case, this parameter should be called baseImage.

You can't snapshot a sanpshot, it makes no sense as it can't change and you will get the same object.

...

userData - optional data that will be attached to the new Snapshot, could be anything that the user desires. options - options to modify VDSMs default behavior

returns the id of the new Snapshot

copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={}) targetRepoId - The ID of a connected repo where the new image will be created imageId - The image you wish to copy

Do we locate the sourceRepoId automatically based on the imageId?

I have written about that below when I talk about common options

...

baseImageId - if specified, the new image will contain only the diff between image and Id. If None the new image will contain all the bits of image Id. This can be used to copy partial parts of images for export. userData - optional data that will be attached to the new image, could be anything that the user desires. options - options to modify VDSMs default behavior

return the Id of the new image. In case of copying an immutable image the ID will be identical to the original image as they contain the same data. However the user should not assume that and always use the value returned from the method.

removeImage(repositoryId, imageId, options={}): repositoryId - The ID of a connected repo where the image to delete resides imageId - The id of the image you wish to delete.

---- getImageStatus(repositoryId, imageId) repositoryId - The ID of a connected repo where the image to check resides imageId - The id of the image you wish to check.

What is in this return value? Is it a single enum indicating whether the image is locked (being copied, etc.) or a list of detailed information (like Volume.getInfo)? (I see some more info below...)

The description below explains everything.

...

All operations return once the operations has been committed to disk NOT when the operation actually completes. This is done so that: - operation come to a stable state as quickly as possible. - In case where there is an SDM, only small portion of the operation actually needs to be performed on the SDM host. - No matter how many times the operation fails and on how many hosts, you can always resume the operation and choose when to do it. - You can stop an operation at any time and remove the resulting object making a distinction between "stop because the host is overloaded" to "I don't want that image"

This means that after calling any operation that creates a new image the user must then call getImageStatus() to check what is the status of the image. The status of the image can be either optimized, degraded, or broken. "Optimized" means that the image is available and you can run VMs of it. "Degraded" means that the image is available and will run VMs but it might be a better way VDSM can represent the underlying data. "Broken" means that the image can't be used at the moment, probably because not all the data has been set up on the volume.

Apart from that VDSM will also return the last persisted status information which will conatin hostID - the last host to try and optimize of fix the image stage - X/Y (eg. 1/10) the last persisted stage of the fix.

Do you have some examples of what the stages would be? I think these should be defined in enums so that the user can check on what the individual stages mean. What happens when the low level implementation of an operation changes? The meaning of the stages will change completely.

No stages can change and are implementation specific, they will be used in the UI to show progress. I added stages because sometimes it's impossible to give percentage for multiple operations. (If you need to do 2 copy operations is each of them 50%?) I agree that we need some way of transferring text. Maybe having another field called current operations. It will be something from a preset enum list: "Copying Image" 1 "Creating Volume" 2 "Changing Metadata" 3 That way we don't commit on number of stages or what they contain but still expose what it is we are doing to display in the UI

...

percent_complete - -1 or 0-100, the last persisted completion percentage of the aforementioned stage. -1 means that no progress is available for that operation.

...

last_error - This will only be filled if the operation failed because of something other then IO or a VDSM crash for obvious reasons. It will usually be set if the task was manually stopped

The user can either be satisfied with that information or as the host specified in host ID if it is still working on that image by checking it's running tasks.

checkStorageRepository(self, repositoryId, options={}): A method to go over a storage repository and scan for any existing problems. This includes degraded\broken images and deleted images that have no yet been physically deleted\merged. It returns a list of Fix objects. Fix objects come in 4 types: clean - cleans data, run them to get more space. optimize - run them to optimize a degraded image

What is an example of a degraded image?

qcow snapshot when the user asked for performance driven image. We create the QCow snapshot first because it will make the VM usable without doing a lot of IO. Optimizing it will be to actually flatten the data and reach the desired state.

...

merge - Merges two images together. Doing this sometimes makes more images ready optimizing or cleaning. The reason it is different from optimize is that unmerged images are considered optimized. mend - mends a broken image

What does this mean?

The image is half copied. Snapshot that we know got corrupted and we have backups of. Whatever operation that moves an image status from broken->(optimized\degraded)

...

The user can read these types and prioritize fixes. Fixes also contain opaque FIX data and they should be sent as received to fixStorageRepository(self, repositoryId, fix, options={}):

That will start a fix operation.

Could we have an automatic fix mode where vdsm just does the right thing (for most things)?

I wanted to have that, but deciding what VDSM should do automatically and when is really difficult and policy driven. The best way to go IMHO is to have policy aware daemons monitor VDSM on the same host and auto-fix according to policy. This shouldn't be part of VDSM as policy changes from manager to manager.

...

All major operations automatically start the appropriate "Fix" to bring the created object to an optimize\degraded state (the one that is quicker) unless one of the options is AutoFix=False. This is only useful for repos that might not be able to create volumes on all hosts (SDM) but would like to have the actual IO distributed in the cluster.

Other common options is the strategy option: It has currently 2 possible values space and performance - In case VDSM has 2 ways of completing the same operation it will tell it to value one over the other. For example, whether to copy all the data or just create a qcow based of a snapshot. The default is space.

I like this a lot.

:)

...

You might have also noticed that it is never explicitly specified where to look for existing images. This is done purposefully, VDSM will always look in all connected repositories for existing objects. For very large setups this might be problematic. To mitigate the problem you have these options: participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to just these repositories and imageHints={imgId: repoId} which will force VDSM to look for those image ID just in those repositories and fail if it doesn't find them there.

I would like to have a better way of specifying these optional parameters without burying them in an options structure. I will think a little more about this. Strategy can just be a two optional flags in a 'flags' argument. For the participatingRepositories and imageHints options, I think we need to use real parameters.

That was what I did in my original design, but then I chose to just move everything to options. The reasoning is that without any options VDSM will try and do what it think is best for the user. Without any options VDSM will use the combination that is most likely to work and in the shortest amount of resources (time\IO) with as little complications as possible. This means space driven operations as they require less IO and space and "automatic source image discovery" as it has the best chance of finding the resources you need. Whatever you put in the options essentially overrides the default behavior for cases where the user is willing to take a risk for some theoretical gain. Using this distinction makes it clear what regular users should use and what expert users should use. Further more, some options might be repository format specific (striping, pre-zeroing) making them part of the interface is problematic. Also, the fact that we know about these options now doesn't mean they should get special treatment, they are still saying "override the default behavior".

-- Adam Litke <agl@us.ibm.com> IBM Linux Technology Center

----- Original Message -----
> From: "Shu Ming" <shuming@linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "engine-devel" <engine-devel@ovirt.org>
> Sent: Thursday, December 6, 2012 11:02:02 AM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> Saggi,
> 
> Thanks for sharing your thought and I get some comments below.
> 
> 
> Saggi Mizrahi:
> > I've been throwing a lot of bits out about the new storage API and
> > I think it's time to talk a bit.
> > I will purposefully try and keep implementation details away and
> > concentrate about how the API looks and how you use it.
> >
> > First major change is in terminology, there is no long a storage
> > domain but a storage repository.
> > This change is done because so many things are already called
> > domain in the system and this will make things less confusing for
> > new-commers with a libvirt background.
> >
> > One other changes is that repositories no longer have a UUID.
> > The UUID was only used in the pool members manifest and is no
> > longer needed.
> >
> >
> > connectStorageRepository(repoId, repoFormat,
> > connectionParameters={}):
> > repoId - is a transient name that will be used to refer to the
> > connected domain, it is not persisted and doesn't have to be the
> > same across the cluster.
> > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > nfs-3.4, clvm-1.2).
> > connectionParameters - This is format specific and will used to
> > tell VDSM how to connect to the repo.
> 
> 
> Where does repoID come from? I think repoID doesn't exist before
> connectStorageRepository() return.  Isn't repoID a return value of
> connectStorageRepository()?
No, repoIDs are no longer part of the domain, they are just a transient handle.
The user can put whatever it wants there as long as it isn't already taken by another currently connected domain.
> 
> >
> > disconnectStorageRepository(self, repoId)
> >
> >
> > In the new API there are only images, some images are mutable and
> > some are not.
> > mutable images are also called VirtualDisks
> > immutable images are also called Snapshots
> >
> > There are no explicit templates, you can create as many images as
> > you want from any snapshot.
> >
> > There are 4 major image operations:
> >
> >
> > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >                    userData={}, options={}):
> >
> > targetRepoId - ID of a connected repo where the disk will be
> > created
> > size - The size of the image you wish to create
> > baseSnapshotId - the ID of the snapshot you want the base the new
> > virtual disk on
> > userData - optional data that will be attached to the new VD, could
> > be anything that the user desires.
> > options - options to modify VDSMs default behavior
> >
> > returns the id of the new VD
> 
> I think we will also need a function to check if a a VirtualDisk is
> based on a specific snapshot.
> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
No, the design is that volume dependencies are an implementation detail.
There is no reason for you to know that an image is physically a snapshot of another.
Logical snapshots, template information, and any other information can be set by the user by using the userData field available for every image.
> 
> >
> > createSnapshot(targetRepoId, baseVirtualDiskId,
> >                 userData={}, options={}):
> > targetRepoId - The ID of a connected repo where the new sanpshot
> > will be created and the original image exists as well.
> > size - The size of the image you wish to create
> > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
> > to snapshot
> > userData - optional data that will be attached to the new Snapshot,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> >
> > returns the id of the new Snapshot
> >
> > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > options={})
> > targetRepoId - The ID of a connected repo where the new image will
> > be created
> > imageId - The image you wish to copy
> > baseImageId - if specified, the new image will contain only the
> > diff between image and Id.
> >                If None the new image will contain all the bits of
> >                image Id. This can be used to copy partial parts of
> >                images for export.
> > userData - optional data that will be attached to the new image,
> > could be anything that the user desires.
> > options - options to modify VDSMs default behavior
> 
> Does this function mean that we can copy the image from one
> repository
> to another repository? Does it cover the semantics of storage
> migration,
> storage backup, storage incremental backup?
Yes, the main purpose is copying to another repo. and you can even do incremental backups.
Also the following flow
1. Run a VM using imageA
2. write to disk
3. Stop VM
4. copy imageA to repoB
5. Run a VM using imageA again
6. Write to disk
7. Stop VM
8. Copy imageA again basing it of imageA_copy1 on repoB creating a diff on repo diff without snapshotting the original image.

> 
> >
> > return the Id of the new image. In case of copying an immutable
> > image the ID will be identical to the original image as they
> > contain the same data. However the user should not assume that and
> > always use the value returned from the method.
> >
> > removeImage(repositoryId, imageId, options={}):
> > repositoryId - The ID of a connected repo where the image to delete
> > resides
> > imageId - The id of the image you wish to delete.
> >
> >
> > ----
> > getImageStatus(repositoryId, imageId)
> > repositoryId - The ID of a connected repo where the image to check
> > resides
> > imageId - The id of the image you wish to check.
> >
> > All operations return once the operations has been committed to
> > disk NOT when the operation actually completes.
> > This is done so that:
> > - operation come to a stable state as quickly as possible.
> > - In case where there is an SDM, only small portion of the
> > operation actually needs to be performed on the SDM host.
> > - No matter how many times the operation fails and on how many
> > hosts, you can always resume the operation and choose when to do
> > it.
> > - You can stop an operation at any time and remove the resulting
> > object making a distinction between "stop because the host is
> > overloaded" to "I don't want that image"
> >
> > This means that after calling any operation that creates a new
> > image the user must then call getImageStatus() to check what is
> > the status of the image.
> > The status of the image can be either optimized, degraded, or
> > broken.
> > "Optimized" means that the image is available and you can run VMs
> > of it.
> > "Degraded" means that the image is available and will run VMs but
> > it might be a better way VDSM can represent the underlying data.
> 
> What does the "represent" mean here?
Anything, but mostly image formate RAW\QCOW2 when performance strategy has been selected.
> > "Broken" means that the image can't be used at the moment, probably
> > because not all the data has been set up on the volume.
> >
> > Apart from that VDSM will also return the last persisted status
> > information which will conatin
> > hostID - the last host to try and optimize of fix the image
> Any host can optimize the image? No need to be SDM?
On anything but lvm based block domains there will not even be an SDM.
On SDM based domains we will try as hard as we can to have as many operations executable on any host.
> 
> > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> > percent_complete - -1 or 0-100, the last persisted completion
> > percentage of the aforementioned stage. -1 means that no progress
> > is available for that operation.
> > last_error - This will only be filled if the operation failed
> > because of something other then IO or a VDSM crash for obvious
> > reasons.
> >               It will usually be set if the task was manually
> >               stopped
> >
> > The user can either be satisfied with that information or as the
> > host specified in host ID if it is still working on that image by
> > checking it's running tasks.
> 
> So we need a function to know what tasks are running on the image
getImageStatus()
> >
> > checkStorageRepository(self, repositoryId, options={}):
> > A method to go over a storage repository and scan for any existing
> > problems. This includes degraded\broken images and deleted images
> > that have no yet been physically deleted\merged.
> > It returns a list of Fix objects.
> > Fix objects come in 4 types:
> > clean - cleans data, run them to get more space.
> > optimize - run them to optimize a degraded image
> > merge - Merges two images together. Doing this sometimes
> >          makes more images ready optimizing or cleaning.
> >          The reason it is different from optimize is that
> >          unmerged images are considered optimized.
> > mend - mends a broken image
> >
> > The user can read these types and prioritize fixes. Fixes also
> > contain opaque FIX data and they should be sent as received to
> > fixStorageRepository(self, repositoryId, fix, options={}):
> >
> > That will start a fix operation.
> >
> >
> > All major operations automatically start the appropriate "Fix" to
> > bring the created object to an optimize\degraded state (the one
> > that is quicker) unless one of the options is
> > AutoFix=False. This is only useful for repos that might not be able
> > to create volumes on all hosts (SDM) but would like to have the
> > actual IO distributed in the cluster.
> >
> > Other common options is the strategy option:
> > It has currently 2 possible values
> > space and performance - In case VDSM has 2 ways of completing the
> > same operation it will tell it to value one over the other. For
> > example, whether to copy all the data or just create a qcow based
> > of a snapshot.
> > The default is space.
> >
> > You might have also noticed that it is never explicitly specified
> > where to look for existing images. This is done purposefully, VDSM
> > will always look in all connected repositories for existing
> > objects.
> > For very large setups this might be problematic. To mitigate the
> > problem you have these options:
> > participatingRepositories=[repoId, ...] which tell VDSM to narrow
> > the search to just these repositories
> > and
> > imageHints={imgId: repoId} which will force VDSM to look for those
> > image ID just in those repositories and fail if it doesn't find
> > them there.
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> 
> --
> ---
> 舒明 Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com or
> shuming@linux.vnet.ibm.com
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> 
> 

于 2012-12-7 13:23, Deepak C Shetty:
> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
>>
>> ----- Original Message -----
>>> From: "Shu Ming" <shuming@linux.vnet.ibm.com>
>>> To: "Saggi Mizrahi" <smizrahi@redhat.com>
>>> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, 
>>> "engine-devel" <engine-devel@ovirt.org>
>>> Sent: Thursday, December 6, 2012 11:02:02 AM
>>> Subject: Re: [vdsm] RFC: New Storage API
>>>
>>> Saggi,
>>>
>>> Thanks for sharing your thought and I get some comments below.
>>>
>>>
>>> Saggi Mizrahi:
>>>> I've been throwing a lot of bits out about the new storage API and
>>>> I think it's time to talk a bit.
>>>> I will purposefully try and keep implementation details away and
>>>> concentrate about how the API looks and how you use it.
>>>>
>>>> First major change is in terminology, there is no long a storage
>>>> domain but a storage repository.
>>>> This change is done because so many things are already called
>>>> domain in the system and this will make things less confusing for
>>>> new-commers with a libvirt background.
>>>>
>>>> One other changes is that repositories no longer have a UUID.
>>>> The UUID was only used in the pool members manifest and is no
>>>> longer needed.
>>>>
>>>>
>>>> connectStorageRepository(repoId, repoFormat,
>>>> connectionParameters={}):
>>>> repoId - is a transient name that will be used to refer to the
>>>> connected domain, it is not persisted and doesn't have to be the
>>>> same across the cluster.
>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
>>>> nfs-3.4, clvm-1.2).
>>>> connectionParameters - This is format specific and will used to
>>>> tell VDSM how to connect to the repo.
>>>
>>> Where does repoID come from? I think repoID doesn't exist before
>>> connectStorageRepository() return.  Isn't repoID a return value of
>>> connectStorageRepository()?
>> No, repoIDs are no longer part of the domain, they are just a 
>> transient handle.
>> The user can put whatever it wants there as long as it isn't already 
>> taken by another currently connected domain.
>
> So what happens when user mistakenly gives a repoID that is in use 
> before.. there should be something in the return value that specifies 
> the error and/or reason for error so that user can try with a new/diff 
> repoID ?

I think let the user to give the repoID is meaningless and error-prune.  
Developer must maintain a a unique ID list for every storage repository 
connected.

>
>>>> disconnectStorageRepository(self, repoId)
>>>>
>>>>
>>>> In the new API there are only images, some images are mutable and
>>>> some are not.
>>>> mutable images are also called VirtualDisks
>>>> immutable images are also called Snapshots
>>>>
>>>> There are no explicit templates, you can create as many images as
>>>> you want from any snapshot.
>>>>
>>>> There are 4 major image operations:
>>>>
>>>>
>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>>>>                     userData={}, options={}):
>>>>
>>>> targetRepoId - ID of a connected repo where the disk will be
>>>> created
>>>> size - The size of the image you wish to create
>>>> baseSnapshotId - the ID of the snapshot you want the base the new
>>>> virtual disk on
>>>> userData - optional data that will be attached to the new VD, could
>>>> be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>
> IIUC, i can use options to do storage offloads ? For eg. I can create 
> a LUN that represents this VD on my storage array based on the 
> 'options' parameter ? Is this the intended way to use 'options' ?
>
>>>>
>>>> returns the id of the new VD
>>> I think we will also need a function to check if a a VirtualDisk is
>>> based on a specific snapshot.
>>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
>> No, the design is that volume dependencies are an implementation detail.
>> There is no reason for you to know that an image is physically a 
>> snapshot of another.
>> Logical snapshots, template information, and any other information 
>> can be set by the user by using the userData field available for 
>> every image.
>>>> createSnapshot(targetRepoId, baseVirtualDiskId,
>>>>                  userData={}, options={}):
>>>> targetRepoId - The ID of a connected repo where the new sanpshot
>>>> will be created and the original image exists as well.
>>>> size - The size of the image you wish to create
>>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
>>>> to snapshot
>>>> userData - optional data that will be attached to the new Snapshot,
>>>> could be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>>>>
>>>> returns the id of the new Snapshot
>>>>
>>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
>>>> options={})
>>>> targetRepoId - The ID of a connected repo where the new image will
>>>> be created
>>>> imageId - The image you wish to copy
>>>> baseImageId - if specified, the new image will contain only the
>>>> diff between image and Id.
>>>>                 If None the new image will contain all the bits of
>>>>                 image Id. This can be used to copy partial parts of
>>>>                 images for export.
>>>> userData - optional data that will be attached to the new image,
>>>> could be anything that the user desires.
>>>> options - options to modify VDSMs default behavior
>>> Does this function mean that we can copy the image from one
>>> repository
>>> to another repository? Does it cover the semantics of storage
>>> migration,
>>> storage backup, storage incremental backup?
>> Yes, the main purpose is copying to another repo. and you can even do 
>> incremental backups.
>> Also the following flow
>> 1. Run a VM using imageA
>> 2. write to disk
>> 3. Stop VM
>> 4. copy imageA to repoB
>> 5. Run a VM using imageA again
>> 6. Write to disk
>> 7. Stop VM
>> 8. Copy imageA again basing it of imageA_copy1 on repoB creating a 
>> diff on repo diff without snapshotting the original image.
>>>> return the Id of the new image. In case of copying an immutable
>>>> image the ID will be identical to the original image as they
>>>> contain the same data. However the user should not assume that and
>>>> always use the value returned from the method.
Do you mean the ID returned by copyImage() will be the same as the 
origin ID?  It is okay for copying to a different repository than the 
original repository because they have different namespace. However, if 
we do the copy inside in the same repository, we should get a new 
different ID.


>>>>
>>>> removeImage(repositoryId, imageId, options={}):
>>>> repositoryId - The ID of a connected repo where the image to delete
>>>> resides
>>>> imageId - The id of the image you wish to delete.
>>>>
>>>>
>>>> ----
>>>> getImageStatus(repositoryId, imageId)
>>>> repositoryId - The ID of a connected repo where the image to check
>>>> resides
>>>> imageId - The id of the image you wish to check.
>>>>
>>>> All operations return once the operations has been committed to
>>>> disk NOT when the operation actually completes.
>>>> This is done so that:
>>>> - operation come to a stable state as quickly as possible.
>>>> - In case where there is an SDM, only small portion of the
>>>> operation actually needs to be performed on the SDM host.
>>>> - No matter how many times the operation fails and on how many
>>>> hosts, you can always resume the operation and choose when to do
>>>> it.
>>>> - You can stop an operation at any time and remove the resulting
>>>> object making a distinction between "stop because the host is
>>>> overloaded" to "I don't want that image"
>>>>
>>>> This means that after calling any operation that creates a new
>>>> image the user must then call getImageStatus() to check what is
>>>> the status of the image.
>>>> The status of the image can be either optimized, degraded, or
>>>> broken.
>>>> "Optimized" means that the image is available and you can run VMs
>>>> of it.
>>>> "Degraded" means that the image is available and will run VMs but
>>>> it might be a better way VDSM can represent the underlying data.
>
> Calling qcow2 based snapshot degraded (meaning its degraded in perf, 
> as its space optimzed ) and calling raw images as optimised ( meaning 
> its optimised for perf as its space in-efficient) is confusing. 
> Degraded sounds like a bad thing when seen by the end-user :) I think 
> there is scope for having some better and less confusing terminology 
> here ?
>
>>> What does the "represent" mean here?
>> Anything, but mostly image formate RAW\QCOW2 when performance 
>> strategy has been selected.
>>>> "Broken" means that the image can't be used at the moment, probably
>>>> because not all the data has been set up on the volume.
>>>>
>>>> Apart from that VDSM will also return the last persisted status
>>>> information which will conatin
>>>> hostID - the last host to try and optimize of fix the image
>>> Any host can optimize the image? No need to be SDM?
>> On anything but lvm based block domains there will not even be an SDM.
>> On SDM based domains we will try as hard as we can to have as many 
>> operations executable on any host.
>
> 1) Can you provide more info on why there is a exception for 'lvm 
> based block domain'. Its not coming out clearly.

I guess Saggi means we can leverage CLVM to build a cluster based LVM 
group and no need to have SDM.  If I understand correctly, in this 
proposal,  we manage different types of the repositories at the same 
type, say, one localFS repository, one iSCSI repository, one nfs 
repository at the same time, also the images can be copied to each other 
back and forth.  When you copy a image from repository without SDM to a 
repository with SDM, we should use the later SDM to execute the copy 
operation.

> 2) Based on the terminology change, domain is now replaced by 
> repository, SDM should now be more aptly called SRM (storage repo 
> manager) so that we are consistent in the usage of terminology
> 3) Can you provide some example flow / scenario to understnad how with 
> and without SDM domains work ? Especially how the disk based lock is 
> taken if no SDM ?
>
>>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
>>>> percent_complete - -1 or 0-100, the last persisted completion
>>>> percentage of the aforementioned stage. -1 means that no progress
>>>> is available for that operation.
>>>> last_error - This will only be filled if the operation failed
>>>> because of something other then IO or a VDSM crash for obvious
>>>> reasons.
>>>>                It will usually be set if the task was manually
>>>>                stopped
>>>>
>>>> The user can either be satisfied with that information or as the
>>>> host specified in host ID if it is still working on that image by
>>>> checking it's running tasks.
>>> So we need a function to know what tasks are running on the image
>> getImageStatus()
>>>> checkStorageRepository(self, repositoryId, options={}):
>>>> A method to go over a storage repository and scan for any existing
>>>> problems. This includes degraded\broken images and deleted images
>>>> that have no yet been physically deleted\merged.
>>>> It returns a list of Fix objects.
>>>> Fix objects come in 4 types:
>>>> clean - cleans data, run them to get more space.
>>>> optimize - run them to optimize a degraded image
>>>> merge - Merges two images together. Doing this sometimes
>>>>           makes more images ready optimizing or cleaning.
>>>>           The reason it is different from optimize is that
>>>>           unmerged images are considered optimized.
>>>> mend - mends a broken image

What is a broken image? Does broken mean corruption?  If it is 
corruption, we may need some recoverable checksum for each image.

>>>>
>>>> The user can read these types and prioritize fixes. Fixes also
>>>> contain opaque FIX data and they should be sent as received to
>>>> fixStorageRepository(self, repositoryId, fix, options={}):
>>>>
>>>> That will start a fix operation.
>
> It would be good if you can provide some example or flow of "fix" 
> operation.
> When and Why would somebody want to do it ?
>
> Does 'Fix' here mean that i move from raw to qcow2 format or 
> vice-versa, or there is more to it ?
>
>>>>
>>>>
>>>> All major operations automatically start the appropriate "Fix" to
>>>> bring the created object to an optimize\degraded state (the one
>>>> that is quicker) unless one of the options is
>>>> AutoFix=False. This is only useful for repos that might not be able
>>>> to create volumes on all hosts (SDM) but would like to have the
>>>> actual IO distributed in the cluster.
>>>>
>>>> Other common options is the strategy option:
>>>> It has currently 2 possible values
>>>> space and performance - In case VDSM has 2 ways of completing the
>>>> same operation it will tell it to value one over the other. For
>>>> example, whether to copy all the data or just create a qcow based
>>>> of a snapshot.
>>>> The default is space.
>>>>
>>>> You might have also noticed that it is never explicitly specified
>>>> where to look for existing images. This is done purposefully, VDSM
>>>> will always look in all connected repositories for existing
>>>> objects.
>>>> For very large setups this might be problematic. To mitigate the
>>>> problem you have these options:
>>>> participatingRepositories=[repoId, ...] which tell VDSM to narrow
>>>> the search to just these repositories
>>>> and
>>>> imageHints={imgId: repoId} which will force VDSM to look for those
>>>> image ID just in those repositories and fail if it doesn't find
>>>> them there.
>>>> _______________________________________________
>>>> vdsm-devel mailing list
>>>> vdsm-devel@lists.fedorahosted.org
>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>>>
>>> -- 
>>> ---
>>> 舒明 Shu Ming
>>> Open Virtualization Engineerning; CSTL, IBM Corp.
>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com or
>>> shuming@linux.vnet.ibm.com
>>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
>>> District, Beijing 100193, PRC
>>>
>>>
>>>
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel@lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>


-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com or shuming@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

----- Original Message -----
> From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development"
> <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com>
> Sent: Friday, December 7, 2012 12:23:15 AM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
> >
> > ----- Original Message -----
> >> From: "Shu Ming" <shuming@linux.vnet.ibm.com>
> >> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> >> Cc: "VDSM Project Development"
> >> <vdsm-devel@lists.fedorahosted.org>, "engine-devel"
> >> <engine-devel@ovirt.org>
> >> Sent: Thursday, December 6, 2012 11:02:02 AM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> Saggi,
> >>
> >> Thanks for sharing your thought and I get some comments below.
> >>
> >>
> >> Saggi Mizrahi:
> >>> I've been throwing a lot of bits out about the new storage API
> >>> and
> >>> I think it's time to talk a bit.
> >>> I will purposefully try and keep implementation details away and
> >>> concentrate about how the API looks and how you use it.
> >>>
> >>> First major change is in terminology, there is no long a storage
> >>> domain but a storage repository.
> >>> This change is done because so many things are already called
> >>> domain in the system and this will make things less confusing for
> >>> new-commers with a libvirt background.
> >>>
> >>> One other changes is that repositories no longer have a UUID.
> >>> The UUID was only used in the pool members manifest and is no
> >>> longer needed.
> >>>
> >>>
> >>> connectStorageRepository(repoId, repoFormat,
> >>> connectionParameters={}):
> >>> repoId - is a transient name that will be used to refer to the
> >>> connected domain, it is not persisted and doesn't have to be the
> >>> same across the cluster.
> >>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>> nfs-3.4, clvm-1.2).
> >>> connectionParameters - This is format specific and will used to
> >>> tell VDSM how to connect to the repo.
> >>
> >> Where does repoID come from? I think repoID doesn't exist before
> >> connectStorageRepository() return.  Isn't repoID a return value of
> >> connectStorageRepository()?
> > No, repoIDs are no longer part of the domain, they are just a
> > transient handle.
> > The user can put whatever it wants there as long as it isn't
> > already taken by another currently connected domain.
> 
> So what happens when user mistakenly gives a repoID that is in use
> before.. there should be something in the return value that specifies
> the error and/or reason for error so that user can try with a
> new/diff
> repoID ?
Asi I said, connect fails if the repoId is in use ATM.
> 
> >>> disconnectStorageRepository(self, repoId)
> >>>
> >>>
> >>> In the new API there are only images, some images are mutable and
> >>> some are not.
> >>> mutable images are also called VirtualDisks
> >>> immutable images are also called Snapshots
> >>>
> >>> There are no explicit templates, you can create as many images as
> >>> you want from any snapshot.
> >>>
> >>> There are 4 major image operations:
> >>>
> >>>
> >>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>                     userData={}, options={}):
> >>>
> >>> targetRepoId - ID of a connected repo where the disk will be
> >>> created
> >>> size - The size of the image you wish to create
> >>> baseSnapshotId - the ID of the snapshot you want the base the new
> >>> virtual disk on
> >>> userData - optional data that will be attached to the new VD,
> >>> could
> >>> be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> 
> IIUC, i can use options to do storage offloads ? For eg. I can create
> a
> LUN that represents this VD on my storage array based on the
> 'options'
> parameter ? Is this the intended way to use 'options' ?
No, this has nothing to do with offloads.
If by "offloads" you mean having other VDSM hosts to the heavy lifting then this is what the option autoFix=False and the fix mechanism is for.
If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible.
In any case, how we manage LUNs (if they are even used) is an implementation detail.
> 
> >>>
> >>> returns the id of the new VD
> >> I think we will also need a function to check if a a VirtualDisk
> >> is
> >> based on a specific snapshot.
> >> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> > No, the design is that volume dependencies are an implementation
> > detail.
> > There is no reason for you to know that an image is physically a
> > snapshot of another.
> > Logical snapshots, template information, and any other information
> > can be set by the user by using the userData field available for
> > every image.
> >>> createSnapshot(targetRepoId, baseVirtualDiskId,
> >>>                  userData={}, options={}):
> >>> targetRepoId - The ID of a connected repo where the new sanpshot
> >>> will be created and the original image exists as well.
> >>> size - The size of the image you wish to create
> >>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> >>> want
> >>> to snapshot
> >>> userData - optional data that will be attached to the new
> >>> Snapshot,
> >>> could be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> >>>
> >>> returns the id of the new Snapshot
> >>>
> >>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> >>> options={})
> >>> targetRepoId - The ID of a connected repo where the new image
> >>> will
> >>> be created
> >>> imageId - The image you wish to copy
> >>> baseImageId - if specified, the new image will contain only the
> >>> diff between image and Id.
> >>>                 If None the new image will contain all the bits
> >>>                 of
> >>>                 image Id. This can be used to copy partial parts
> >>>                 of
> >>>                 images for export.
> >>> userData - optional data that will be attached to the new image,
> >>> could be anything that the user desires.
> >>> options - options to modify VDSMs default behavior
> >> Does this function mean that we can copy the image from one
> >> repository
> >> to another repository? Does it cover the semantics of storage
> >> migration,
> >> storage backup, storage incremental backup?
> > Yes, the main purpose is copying to another repo. and you can even
> > do incremental backups.
> > Also the following flow
> > 1. Run a VM using imageA
> > 2. write to disk
> > 3. Stop VM
> > 4. copy imageA to repoB
> > 5. Run a VM using imageA again
> > 6. Write to disk
> > 7. Stop VM
> > 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
> > diff on repo diff without snapshotting the original image.
> >
> >>> return the Id of the new image. In case of copying an immutable
> >>> image the ID will be identical to the original image as they
> >>> contain the same data. However the user should not assume that
> >>> and
> >>> always use the value returned from the method.
> >>>
> >>> removeImage(repositoryId, imageId, options={}):
> >>> repositoryId - The ID of a connected repo where the image to
> >>> delete
> >>> resides
> >>> imageId - The id of the image you wish to delete.
> >>>
> >>>
> >>> ----
> >>> getImageStatus(repositoryId, imageId)
> >>> repositoryId - The ID of a connected repo where the image to
> >>> check
> >>> resides
> >>> imageId - The id of the image you wish to check.
> >>>
> >>> All operations return once the operations has been committed to
> >>> disk NOT when the operation actually completes.
> >>> This is done so that:
> >>> - operation come to a stable state as quickly as possible.
> >>> - In case where there is an SDM, only small portion of the
> >>> operation actually needs to be performed on the SDM host.
> >>> - No matter how many times the operation fails and on how many
> >>> hosts, you can always resume the operation and choose when to do
> >>> it.
> >>> - You can stop an operation at any time and remove the resulting
> >>> object making a distinction between "stop because the host is
> >>> overloaded" to "I don't want that image"
> >>>
> >>> This means that after calling any operation that creates a new
> >>> image the user must then call getImageStatus() to check what is
> >>> the status of the image.
> >>> The status of the image can be either optimized, degraded, or
> >>> broken.
> >>> "Optimized" means that the image is available and you can run VMs
> >>> of it.
> >>> "Degraded" means that the image is available and will run VMs but
> >>> it might be a better way VDSM can represent the underlying data.
> 
> Calling qcow2 based snapshot degraded (meaning its degraded in perf,
> as
> its space optimzed ) and calling raw images as optimised ( meaning
> its
> optimised for perf as its space in-efficient) is confusing. Degraded
> sounds like a bad thing when seen by the end-user :) I think there is
> scope for having some better and less confusing terminology here ?
No, that is not what I meant. If the user asked for a space driven image then the qcow2 version is the optimized version.
I chose my words carefully when I originally wrote "a better way VDSM can represent the underlying data".
"better" can mean different things in different circumstances.
How VDSM makes it "better" is an implementation detail. It can even change between VDSM versions.
An extreme case is if we ever decide to make QED the preferred format.
That could potentially suddenly mark all images that are internally represented as qcow2 as degraded because VDSM has found a "better way to represent the data".
All the user has to know is that "degraded images" can be used but they are in a what VDSM considers to be a sub-optimal state.
It's up to the user to decide whether to optimize now, later or never.
> 
> >> What does the "represent" mean here?
> > Anything, but mostly image formate RAW\QCOW2 when performance
> > strategy has been selected.
> >>> "Broken" means that the image can't be used at the moment,
> >>> probably
> >>> because not all the data has been set up on the volume.
> >>>
> >>> Apart from that VDSM will also return the last persisted status
> >>> information which will conatin
> >>> hostID - the last host to try and optimize of fix the image
> >> Any host can optimize the image? No need to be SDM?
> > On anything but lvm based block domains there will not even be an
> > SDM.
> > On SDM based domains we will try as hard as we can to have as many
> > operations executable on any host.
> 
> 1) Can you provide more info on why there is a exception for 'lvm
> based
> block domain'. Its not coming out clearly.
File based domains are responsible for syncing up object manipulation (creation\deletion)
The backend is responsible for making sure it all works either by having a single writer (NFS) or having it's own locking mechanism (gluster).
In our LVM based domains VDSM is responsible for basic object manipulation.
The current design uses an approach where there is a single host responsible for object creation\deleteion it is the SRM\SDM\SPM\S?M.
If we ever find a way to make it fully clustered without a big hit in performance the S?M requirement will be removed form that type of domain.
> 2) Based on the terminology change, domain is now replaced by
> repository, SDM should now be more aptly called SRM (storage repo
> manager) so that we are consistent in the usage of terminology
OK
> 3) Can you provide some example flow / scenario to understnad how
> with
> and without SDM domains work ? Especially how the disk based lock is
> taken if no SDM ?
This is part of the repo API which is out of the scope of this document but in general terms.
getStorageRepositoryContraints() will return the NEED_S?M flag which will notify the manager that it needs to elect an S?M for this repository instance.
All image createVirtualDisk() createSnapshot() and copyImage() and some fixes() will need to be performed on the S?M, removeImage() and some of the fixes could be performed on any host.
As previously noted sending the option autoFix=False will make VDSM not automatically start a fix operation after the operation finished persisting the request and creating the object.
This means you could offload the actual date IO to another host.
Because what fixes can run on what host can change between versions you will just have to try and see if VDSM rejects it on hosts other then the SPM.

There are also plans on abstracting further and making hosts that are not the S?M ask the S?M to create\delete objects for them so you can run any request on any host.
I would very much like to implement that but I am still not sure how efficient inter VDSM communication is to commit to that.
This can be easily implemented in the future by a new flag that marks that even though this repo requires an S?M it doesn't matter where you perform the operation.
> 
> >>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> >>> percent_complete - -1 or 0-100, the last persisted completion
> >>> percentage of the aforementioned stage. -1 means that no progress
> >>> is available for that operation.
> >>> last_error - This will only be filled if the operation failed
> >>> because of something other then IO or a VDSM crash for obvious
> >>> reasons.
> >>>                It will usually be set if the task was manually
> >>>                stopped
> >>>
> >>> The user can either be satisfied with that information or as the
> >>> host specified in host ID if it is still working on that image by
> >>> checking it's running tasks.
> >> So we need a function to know what tasks are running on the image
> > getImageStatus()
> >>> checkStorageRepository(self, repositoryId, options={}):
> >>> A method to go over a storage repository and scan for any
> >>> existing
> >>> problems. This includes degraded\broken images and deleted images
> >>> that have no yet been physically deleted\merged.
> >>> It returns a list of Fix objects.
> >>> Fix objects come in 4 types:
> >>> clean - cleans data, run them to get more space.
> >>> optimize - run them to optimize a degraded image
> >>> merge - Merges two images together. Doing this sometimes
> >>>           makes more images ready optimizing or cleaning.
> >>>           The reason it is different from optimize is that
> >>>           unmerged images are considered optimized.
> >>> mend - mends a broken image
> >>>
> >>> The user can read these types and prioritize fixes. Fixes also
> >>> contain opaque FIX data and they should be sent as received to
> >>> fixStorageRepository(self, repositoryId, fix, options={}):
> >>>
> >>> That will start a fix operation.
> 
> It would be good if you can provide some example or flow of "fix"
> operation.
> When and Why would somebody want to do it ?
You would want to do it to get your images to a better state broken->(degraded|optimized) or degraded->optimized.
Getting an image to a functional (degraded|optimized) state is obviously beneficial and getting to an optimized is better by definition.
As an example.
The user called copyImage() that created the all the objects involved and persisted the command and returned.
The snapshot is not yet ready as the actual copy of the data didn't really happen.
Lets say that the method was called with autoFix=False so you now have a broken image (the new image).
You can't use it at the moment. Running a Fix will continue the actual operation moving it to a better state.
VDSM will choose whatever is more likely to get the image to a functional state faster. This means that the image might find itself in a degraded state.
This is good because if the user want to run the VM ASAP it can now do it for the price of performance.
If the user has resource to spare (time\IO\hosts) it can choose to perform another fix and get the image to an optimized state.
This ensures that the images is in the best state according to the users options.
This of course doesn't mean best performance as the user might ask that VDSM value other things over performance.

This system means that you can allocate and deallocate resources at will with minimal consequences.
In some cases advanced users can use this system in conjunction with live merge \ live migration to actually "optimize" while it's running.

This system means that VDSM can change it's policy and behavior between versions without that breaking the interface.
We find a way to skip degraded state for some use case, great.
We found a way to make some use case finish faster but in degraded mode, awesome.
We found a way to make some operations run on all hosts, fantastic.
The user can always handle it because it never assumes the physical result of any operation just the logical.

> 
> Does 'Fix' here mean that i move from raw to qcow2 format or
> vice-versa,
> or there is more to it ?
Fix could literally mean anything, format conversion, compression\decompression, cleanups, merges.
> 
> >>>
> >>>
> >>> All major operations automatically start the appropriate "Fix" to
> >>> bring the created object to an optimize\degraded state (the one
> >>> that is quicker) unless one of the options is
> >>> AutoFix=False. This is only useful for repos that might not be
> >>> able
> >>> to create volumes on all hosts (SDM) but would like to have the
> >>> actual IO distributed in the cluster.
> >>>
> >>> Other common options is the strategy option:
> >>> It has currently 2 possible values
> >>> space and performance - In case VDSM has 2 ways of completing the
> >>> same operation it will tell it to value one over the other. For
> >>> example, whether to copy all the data or just create a qcow based
> >>> of a snapshot.
> >>> The default is space.
> >>>
> >>> You might have also noticed that it is never explicitly specified
> >>> where to look for existing images. This is done purposefully,
> >>> VDSM
> >>> will always look in all connected repositories for existing
> >>> objects.
> >>> For very large setups this might be problematic. To mitigate the
> >>> problem you have these options:
> >>> participatingRepositories=[repoId, ...] which tell VDSM to narrow
> >>> the search to just these repositories
> >>> and
> >>> imageHints={imgId: repoId} which will force VDSM to look for
> >>> those
> >>> image ID just in those repositories and fail if it doesn't find
> >>> them there.
> >>> _______________________________________________
> >>> vdsm-devel mailing list
> >>> vdsm-devel@lists.fedorahosted.org
> >>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>
> >> --
> >> ---
> >> 舒明 Shu Ming
> >> Open Virtualization Engineerning; CSTL, IBM Corp.
> >> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com
> >> or
> >> shuming@linux.vnet.ibm.com
> >> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> >> District, Beijing 100193, PRC
> >>
> >>
> >>
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
>

----- Original Message -----

From: "Adam Litke" <agl@us.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org> Sent: Monday, December 10, 2012 1:49:31 PM Subject: Re: [vdsm] RFC: New Storage API

On Fri, Dec 07, 2012 at 02:53:41PM -0500, Saggi Mizrahi wrote:

<snip>

...

...

1) Can you provide more info on why there is a exception for 'lvm based block domain'. Its not coming out clearly. File based domains are responsible for syncing up object manipulation (creation\deletion) The backend is responsible for making sure it all works either by having a single writer (NFS) or having it's own locking mechanism (gluster). In our LVM based domains VDSM is responsible for basic object manipulation. The current design uses an approach where there is a single host responsible for object creation\deleteion it is the SRM\SDM\SPM\S?M. If we ever find a way to make it fully clustered without a big hit in performance the S?M requirement will be removed form that type of domain.

I would like to see us maintain a LOCALFS domain as well. For this, we would also need SRM, correct? No, why?

-- Adam Litke <agl@us.ibm.com> IBM Linux Technology Center

On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:

----- Original Message -----

...

From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Friday, December 7, 2012 12:23:15 AM Subject: Re: [vdsm] RFC: New Storage API

...

...

From: "Shu Ming" <shuming@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "engine-devel" <engine-devel@ovirt.org> Sent: Thursday, December 6, 2012 11:02:02 AM Subject: Re: [vdsm] RFC: New Storage API

Saggi,

Thanks for sharing your thought and I get some comments below.

Saggi Mizrahi:

...

I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit. I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a storage domain but a storage repository. This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID. The UUID was only used in the pool members manifest and is no longer needed.

connectStorageRepository(repoId, repoFormat, connectionParameters={}): repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster. repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2). connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo. Where does repoID come from? I think repoID doesn't exist before connectStorageRepository() return. Isn't repoID a return value of connectStorageRepository()? No, repoIDs are no longer part of the domain, they are just a

----- Original Message ----- transient handle. The user can put whatever it wants there as long as it isn't already taken by another currently connected domain. So what happens when user mistakenly gives a repoID that is in use before.. there should be something in the return value that specifies

...

...

...

disconnectStorageRepository(self, repoId)

In the new API there are only images, some images are mutable and some are not. mutable images are also called VirtualDisks immutable images are also called Snapshots

There are no explicit templates, you can create as many images as you want from any snapshot.

There are 4 major image operations:

createVirtualDisk(targetRepoId, size, baseSnapshotId=None, userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be created size - The size of the image you wish to create baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on userData - optional data that will be attached to the new VD, could be anything that the user desires. options - options to modify VDSMs default behavior IIUC, i can use options to do storage offloads ? For eg. I can create a LUN that represents this VD on my storage array based on the 'options'

On 12/06/2012 10:22 PM, Saggi Mizrahi wrote: the error and/or reason for error so that user can try with a new/diff repoID ? Asi I said, connect fails if the repoId is in use ATM. parameter ? Is this the intended way to use 'options' ? No, this has nothing to do with offloads. If by "offloads" you mean having other VDSM hosts to the heavy lifting then this is what the option autoFix=False and the fix mechanism is for. If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible. In any case, how we manage LUNs (if they are even used) is an implementation detail.

I am a bit more interested in how storage array offloads ( by that I mean, offload VD creation, snapshot, clone etc to the storage array when available/possible) can be done from VDSM ? In the past there were talks of using libSM to do that. How does that strategy play in this new Storage API scenario ? I agree its implmn detail, but how & where does that implm sit and how it would be triggered is not very clear to me. Looking at createVD args, it sounded like 'options' seems to be a trigger point for deciding whether to use storage offloads or not, but you spoke otherwise :) Can you provide your vision on how VDSM can understand the storage array capabilities & exploit storgae array offloads in this New Storage API context ? -- Thanks deepak

----- Original Message -----

...

From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Sunday, December 16, 2012 11:40:01 PM Subject: Re: [vdsm] RFC: New Storage API

On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:

...

...

From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Friday, December 7, 2012 12:23:15 AM Subject: Re: [vdsm] RFC: New Storage API

...

...

From: "Shu Ming" <shuming@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "engine-devel" <engine-devel@ovirt.org> Sent: Thursday, December 6, 2012 11:02:02 AM Subject: Re: [vdsm] RFC: New Storage API

Saggi,

Thanks for sharing your thought and I get some comments below.

Saggi Mizrahi: > I've been throwing a lot of bits out about the new storage > API > and > I think it's time to talk a bit. > I will purposefully try and keep implementation details away > and > concentrate about how the API looks and how you use it. > > First major change is in terminology, there is no long a > storage > domain but a storage repository. > This change is done because so many things are already called > domain in the system and this will make things less confusing > for > new-commers with a libvirt background. > > One other changes is that repositories no longer have a UUID. > The UUID was only used in the pool members manifest and is no > longer needed. > > > connectStorageRepository(repoId, repoFormat, > connectionParameters={}): > repoId - is a transient name that will be used to refer to > the > connected domain, it is not persisted and doesn't have to be > the > same across the cluster. > repoFormat - Similar to what used to be type (eg. > localfs-1.0, > nfs-3.4, clvm-1.2). > connectionParameters - This is format specific and will used > to > tell VDSM how to connect to the repo. Where does repoID come from? I think repoID doesn't exist before connectStorageRepository() return. Isn't repoID a return value of connectStorageRepository()? No, repoIDs are no longer part of the domain, they are just a

----- Original Message ----- transient handle. The user can put whatever it wants there as long as it isn't already taken by another currently connected domain. So what happens when user mistakenly gives a repoID that is in use before.. there should be something in the return value that specifies

...

...

> disconnectStorageRepository(self, repoId) > > > In the new API there are only images, some images are mutable > and > some are not. > mutable images are also called VirtualDisks > immutable images are also called Snapshots > > There are no explicit templates, you can create as many > images > as > you want from any snapshot. > > There are 4 major image operations: > > > createVirtualDisk(targetRepoId, size, baseSnapshotId=None, > userData={}, options={}): > > targetRepoId - ID of a connected repo where the disk will be > created > size - The size of the image you wish to create > baseSnapshotId - the ID of the snapshot you want the base the > new > virtual disk on > userData - optional data that will be attached to the new VD, > could > be anything that the user desires. > options - options to modify VDSMs default behavior IIUC, i can use options to do storage offloads ? For eg. I can create a LUN that represents this VD on my storage array based on the 'options'

On 12/06/2012 10:22 PM, Saggi Mizrahi wrote: the error and/or reason for error so that user can try with a new/diff repoID ? Asi I said, connect fails if the repoId is in use ATM. parameter ? Is this the intended way to use 'options' ? No, this has nothing to do with offloads. If by "offloads" you mean having other VDSM hosts to the heavy

----- Original Message ----- lifting then this is what the option autoFix=False and the fix mechanism is for. If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible. In any case, how we manage LUNs (if they are even used) is an implementation detail.

I am a bit more interested in how storage array offloads ( by that I mean, offload VD creation, snapshot, clone etc to the storage array when available/possible) can be done from VDSM ? In the past there were talks of using libSM to do that. How does that strategy play in this new Storage API scenario ? I agree its implmn detail, but how & where does that implm sit and how it would be triggered is not very clear to me. Looking at createVD args, it sounded like 'options' seems to be a trigger point for deciding whether to use storage offloads or not, but you spoke otherwise :) Can you provide your vision on how VDSM can understand the storage array capabilities & exploit storgae array offloads in this New Storage API context ? -- Thanks deepak Some will be used automatically whenever possible (storage offloading). Features that favor a specific strategy will be activated when the

----- Original Message ----- proper strategy (space, performance) option is selected. In cases when only the user can know whether to use a feature or not we will have options to turn that on. In any case every domain exports a capability list through GetRepositoryCapabilities() that returns a list off repository capabilities. Some capabilities are VDSM specific like CLUSTERED or REQUIRES_SRM. Some are storage capabilities like NATIVE_SNAPSHOTS, NATIVE_THIN_PROVISIONING, SPARSE_VOLUMES, etc...

We are also considering an override mechanism where you can disable features in storage that supports it by setting it in the domain options. This will be done with NO_XXXXX (eg. NO_NATIVE_SNAPSHOTS). This will make the domain not use or expose the capability through the API. I assume it will only be used for testing or in cases where the storage array is known to have problems with a certain feature. Not everything can be disables as an example there is no real way to disable NATIVE_THING_PROVISIONING or SPARSE_VOLUMES.

Saggi, there are several different discussions going on here which I think require some clearing up (and perhaps splitting). what I think is missing here: 1. distinction of what we believe should be at repo level and what at disk level e.g. pre/postZero at repo level, native snapshots as described above would also be repo level (not defined per disk) etc. 2. how/where storage offload would work? is there a single implementation for repos which detects automatically storage capabilities or repo class for each storage type 3. and biggest topic is probably - a mapping of the image operations and details about the flows (did you send something about that?) - i.e. create vdisk flow, copy, etc.

...

_______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

----- Original Message -----

...

From: "Ayal Baron" <abaron@redhat.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Monday, January 14, 2013 4:56:05 PM Subject: Re: [vdsm] RFC: New Storage API

----- Original Message -----

...

...

From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Sunday, December 16, 2012 11:40:01 PM Subject: Re: [vdsm] RFC: New Storage API

On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:

...

...

From: "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "Deepak C Shetty" <deepakcs@linux.vnet.ibm.com> Sent: Friday, December 7, 2012 12:23:15 AM Subject: Re: [vdsm] RFC: New Storage API

On 12/06/2012 10:22 PM, Saggi Mizrahi wrote: > ----- Original Message ----- >> From: "Shu Ming" <shuming@linux.vnet.ibm.com> >> To: "Saggi Mizrahi" <smizrahi@redhat.com> >> Cc: "VDSM Project Development" >> <vdsm-devel@lists.fedorahosted.org>, "engine-devel" >> <engine-devel@ovirt.org> >> Sent: Thursday, December 6, 2012 11:02:02 AM >> Subject: Re: [vdsm] RFC: New Storage API >> >> Saggi, >> >> Thanks for sharing your thought and I get some comments >> below. >> >> >> Saggi Mizrahi: >>> I've been throwing a lot of bits out about the new >>> storage >>> API >>> and >>> I think it's time to talk a bit. >>> I will purposefully try and keep implementation details >>> away >>> and >>> concentrate about how the API looks and how you use it. >>> >>> First major change is in terminology, there is no long a >>> storage >>> domain but a storage repository. >>> This change is done because so many things are already >>> called >>> domain in the system and this will make things less >>> confusing >>> for >>> new-commers with a libvirt background. >>> >>> One other changes is that repositories no longer have a >>> UUID. >>> The UUID was only used in the pool members manifest and >>> is >>> no >>> longer needed. >>> >>> >>> connectStorageRepository(repoId, repoFormat, >>> connectionParameters={}): >>> repoId - is a transient name that will be used to refer >>> to >>> the >>> connected domain, it is not persisted and doesn't have to >>> be >>> the >>> same across the cluster. >>> repoFormat - Similar to what used to be type (eg. >>> localfs-1.0, >>> nfs-3.4, clvm-1.2). >>> connectionParameters - This is format specific and will >>> used >>> to >>> tell VDSM how to connect to the repo. >> Where does repoID come from? I think repoID doesn't exist >> before >> connectStorageRepository() return. Isn't repoID a return >> value >> of >> connectStorageRepository()? > No, repoIDs are no longer part of the domain, they are just > a > transient handle. > The user can put whatever it wants there as long as it > isn't > already taken by another currently connected domain. So what happens when user mistakenly gives a repoID that is in use before.. there should be something in the return value that specifies the error and/or reason for error so that user can try with a new/diff repoID ? Asi I said, connect fails if the repoId is in use ATM. >>> disconnectStorageRepository(self, repoId) >>> >>> >>> In the new API there are only images, some images are >>> mutable >>> and >>> some are not. >>> mutable images are also called VirtualDisks >>> immutable images are also called Snapshots >>> >>> There are no explicit templates, you can create as many >>> images >>> as >>> you want from any snapshot. >>> >>> There are 4 major image operations: >>> >>> >>> createVirtualDisk(targetRepoId, size, >>> baseSnapshotId=None, >>> userData={}, options={}): >>> >>> targetRepoId - ID of a connected repo where the disk will >>> be >>> created >>> size - The size of the image you wish to create >>> baseSnapshotId - the ID of the snapshot you want the base >>> the >>> new >>> virtual disk on >>> userData - optional data that will be attached to the new >>> VD, >>> could >>> be anything that the user desires. >>> options - options to modify VDSMs default behavior IIUC, i can use options to do storage offloads ? For eg. I can create a LUN that represents this VD on my storage array based on the 'options' parameter ? Is this the intended way to use 'options' ? No, this has nothing to do with offloads. If by "offloads" you mean having other VDSM hosts to the heavy

----- Original Message ----- lifting then this is what the option autoFix=False and the fix mechanism is for. If you are talking about advanced scsi features (ie. write same) they will be used automatically whenever possible. In any case, how we manage LUNs (if they are even used) is an implementation detail.

I am a bit more interested in how storage array offloads ( by that I mean, offload VD creation, snapshot, clone etc to the storage array when available/possible) can be done from VDSM ? In the past there were talks of using libSM to do that. How does that strategy play in this new Storage API scenario ? I agree its implmn detail, but how & where does that implm sit and how it would be triggered is not very clear to me. Looking at createVD args, it sounded like 'options' seems to be a trigger point for deciding whether to use storage offloads or not, but you spoke otherwise :) Can you provide your vision on how VDSM can understand the storage array capabilities & exploit storgae array offloads in this New Storage API context ? -- Thanks deepak Some will be used automatically whenever possible (storage offloading). Features that favor a specific strategy will be activated when

----- Original Message ----- the proper strategy (space, performance) option is selected. In cases when only the user can know whether to use a feature or not we will have options to turn that on. In any case every domain exports a capability list through GetRepositoryCapabilities() that returns a list off repository capabilities. Some capabilities are VDSM specific like CLUSTERED or REQUIRES_SRM. Some are storage capabilities like NATIVE_SNAPSHOTS, NATIVE_THIN_PROVISIONING, SPARSE_VOLUMES, etc...

We are also considering an override mechanism where you can disable features in storage that supports it by setting it in the domain options. This will be done with NO_XXXXX (eg. NO_NATIVE_SNAPSHOTS). This will make the domain not use or expose the capability through the API. I assume it will only be used for testing or in cases where the storage array is known to have problems with a certain feature. Not everything can be disables as an example there is no real way to disable NATIVE_THING_PROVISIONING or SPARSE_VOLUMES.

Saggi, there are several different discussions going on here which I think require some clearing up (and perhaps splitting). what I think is missing here: 1. distinction of what we believe should be at repo level and what at disk level e.g. pre/postZero at repo level, native snapshots as described above would also be repo level (not defined per disk) etc. No option is inherently repo or disk based (even the ones you mantioned). It all depends on the VDSM version you are interacting with. I am open to

----- Original Message ----- the suggestion on having a way to query whether certain options are supported by what operation and in what level dynamically.

As for post-zeroing, currently it is kind of insulting to our users that we push it as a solution to prevent new VMs from reading old VMs data. It is just not what it does. It will be replaced with pre-zeroing.

The only reason I see for post zeroing is the "wipe HDD before selling it off" use case but I don't see how it is valid in virtualized environments especially with enterprise storage.

I have 2 problems with this: 1. you're assuming 'enterprise' storage and in cloud this is many times not the case at all. On local storage it's pretty safe to assume that what you see is what you get (on block devices at least). 2. we have requests to in fact add additional wiping algorithms to really clean things up exactly for the 'wipe HDD before selling it off' scenario (only valid of course if we actually write on the relevant sectors which is something we cannot guarantee, but with the right disclaimers, this should be fine). Rethinking this, at most post-zero is wasteful, but storage array should guarantee that next time data is read from disk then zeros are served (if it doesn't then it's a security flaw on the storage side which we shouldn't care about).

...

2. how/where storage offload would work? is there a single implementation for repos which detects automatically storage capabilities or repo class for each storage type They will be used automatically if available and supported by VDSM and the repo format. It doesn't matter to this section of the API whether the user will have to manually create repositories in specific formats or if it will be automatic. As it seems, it will probably be automatically. 3. and biggest topic is probably - a mapping of the image operations and details about the flows (did you send something about that?) - i.e. create vdisk flow, copy, etc. The flows you suggest here are basic flows supported with a single API call.

Because of the nature of image states users need to have a mechanism to track said image states and make sure fixes are running (if desired)

Everyone is welcome to give it a shot and see how they can implement high level flows using this API in their own systems. I don't mind helping people figure it out, just ask.

Actually I meant the new images API flows here (i.e. the internals, not how to use it).

...

...

...

_______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

On 12/07/2012 09:53 PM, Saggi Mizrahi wrote:

...

...

...

...

...

connectStorageRepository(repoId, repoFormat, connectionParameters={}): repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster. repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2). connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.

Where does repoID come from? I think repoID doesn't exist before connectStorageRepository() return. Isn't repoID a return value of connectStorageRepository()? No, repoIDs are no longer part of the domain, they are just a transient handle. The user can put whatever it wants there as long as it isn't already taken by another currently connected domain.

So what happens when user mistakenly gives a repoID that is in use before.. there should be something in the return value that specifies the error and/or reason for error so that user can try with a new/diff repoID ? Asi I said, connect fails if the repoId is in use ATM.

so how do add connections to the repo without first disconnecting it ("extend storage domain" flow)?

----- Original Message -----
> From: "Adam Litke" <agl@us.ibm.com>
> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development"
> <vdsm-devel@lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 1:39:51 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Shu Ming" <shuming@linux.vnet.ibm.com>
> > > To: "Saggi Mizrahi" <smizrahi@redhat.com>
> > > Cc: "VDSM Project Development"
> > > <vdsm-devel@lists.fedorahosted.org>, "engine-devel"
> > > <engine-devel@ovirt.org>
> > > Sent: Thursday, December 6, 2012 11:02:02 AM
> > > Subject: Re: [vdsm] RFC: New Storage API
> > > 
> > > Saggi,
> > > 
> > > Thanks for sharing your thought and I get some comments below.
> > > 
> > > 
> > > Saggi Mizrahi:
> > > > I've been throwing a lot of bits out about the new storage API
> > > > and
> > > > I think it's time to talk a bit.
> > > > I will purposefully try and keep implementation details away
> > > > and
> > > > concentrate about how the API looks and how you use it.
> > > >
> > > > First major change is in terminology, there is no long a
> > > > storage
> > > > domain but a storage repository.
> > > > This change is done because so many things are already called
> > > > domain in the system and this will make things less confusing
> > > > for
> > > > new-commers with a libvirt background.
> > > >
> > > > One other changes is that repositories no longer have a UUID.
> > > > The UUID was only used in the pool members manifest and is no
> > > > longer needed.
> > > >
> > > >
> > > > connectStorageRepository(repoId, repoFormat,
> > > > connectionParameters={}):
> > > > repoId - is a transient name that will be used to refer to the
> > > > connected domain, it is not persisted and doesn't have to be
> > > > the
> > > > same across the cluster.
> > > > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > > > nfs-3.4, clvm-1.2).
> > > > connectionParameters - This is format specific and will used to
> > > > tell VDSM how to connect to the repo.
> > > 
> > > 
> > > Where does repoID come from? I think repoID doesn't exist before
> > > connectStorageRepository() return.  Isn't repoID a return value
> > > of
> > > connectStorageRepository()?
> > No, repoIDs are no longer part of the domain, they are just a
> > transient handle.
> > The user can put whatever it wants there as long as it isn't
> > already taken by another currently connected domain.
> > > 
> > > >
> > > > disconnectStorageRepository(self, repoId)
> > > >
> > > >
> > > > In the new API there are only images, some images are mutable
> > > > and
> > > > some are not.
> > > > mutable images are also called VirtualDisks
> > > > immutable images are also called Snapshots
> > > >
> > > > There are no explicit templates, you can create as many images
> > > > as
> > > > you want from any snapshot.
> > > >
> > > > There are 4 major image operations:
> > > >
> > > >
> > > > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> > > >                    userData={}, options={}):
> > > >
> > > > targetRepoId - ID of a connected repo where the disk will be
> > > > created
> > > > size - The size of the image you wish to create
> > > > baseSnapshotId - the ID of the snapshot you want the base the
> > > > new
> > > > virtual disk on
> > > > userData - optional data that will be attached to the new VD,
> > > > could
> > > > be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > >
> > > > returns the id of the new VD
> > > 
> > > I think we will also need a function to check if a a VirtualDisk
> > > is
> > > based on a specific snapshot.
> > > Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> > No, the design is that volume dependencies are an implementation
> > detail.
> > There is no reason for you to know that an image is physically a
> > snapshot of another.
> > Logical snapshots, template information, and any other information
> > can be set by the user by using the userData field available for
> > every image.
> 
> Statements like this make me start to worry about your userData
> concept.  It's a
> sign of a bad API if the user needs to invent a custom metadata
> scheme for
> itself.  This reminds me of the abomination that is the 'custom'
> property in the
> vm definition today.
In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.

userData being a "void*" is quite common and I don't understand why you would thing it's a sign of a bad API.
Further more, giving the user choice about how to represent it's own metadata and what fields it want to keep seems reasonable to me.
Especially given the fact that VDSM never reads it.

The reason we are pulling away from the current system of VDSM understanding the extra data is that it makes that data tied to VDSMs on disk format.
VDSM on disk format has to be very stable because of clusters with multiple VDSM versions.
Further more, since this is actually manager data it has to be tied to the manager backward compatibility lifetime as well.
Having it be opaque to VDSM ties it to only one, simpler, support lifetime instead of two.

I guess you are implying that it will make it problematic for multiple users to read userData left by another user because the formats might not be compatible.
The solution is that all parties interested in using VDSM storage agree on format, and common fields, and supportability, and all the other things that choosing a supporting *something* entails.
This is, however, out of the scope of VDSM. When the time comes I think how the userData blob is actually parsed and what fields it keeps should be discussed on ovirt-devel or engine-devel.

The crux of the issue is that VDSM manages only what it cares about and the user can't modify directly.
This is done because everything we expose we commit to.
If you want any information persisted like:
- Human readable name (in whatever encoding)
- Is this a template or a snapshot
- What user owns this image

You can just put it in the userData.
VDSM is not going to impose what encoding you use.
It's not going to decide if you represent your users as IDs or names or ldap queries or Public Keys.
It's not going to decide if you have explicit templates or not.
It's not going to decide if you care what is the logical image chain.
It's not going to decide anything that is out of it's scope.
No format is future proof, no selection of fields will be good for any situation.
I'd much rather it be someone else's problem when any of them need to be changed.
They have currently been VDSMs problem and it has been hell to maintain.

> 
> > > > createSnapshot(targetRepoId, baseVirtualDiskId,
> > > >                 userData={}, options={}):
> > > > targetRepoId - The ID of a connected repo where the new
> > > > sanpshot
> > > > will be created and the original image exists as well.
> > > > size - The size of the image you wish to create
> > > > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> > > > want
> > > > to snapshot
> > > > userData - optional data that will be attached to the new
> > > > Snapshot,
> > > > could be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > >
> > > > returns the id of the new Snapshot
> > > >
> > > > copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> > > > options={})
> > > > targetRepoId - The ID of a connected repo where the new image
> > > > will
> > > > be created
> > > > imageId - The image you wish to copy
> > > > baseImageId - if specified, the new image will contain only the
> > > > diff between image and Id.
> > > >                If None the new image will contain all the bits
> > > >                of
> > > >                image Id. This can be used to copy partial parts
> > > >                of
> > > >                images for export.
> > > > userData - optional data that will be attached to the new
> > > > image,
> > > > could be anything that the user desires.
> > > > options - options to modify VDSMs default behavior
> > > 
> > > Does this function mean that we can copy the image from one
> > > repository
> > > to another repository? Does it cover the semantics of storage
> > > migration,
> > > storage backup, storage incremental backup?
> > Yes, the main purpose is copying to another repo. and you can even
> > do incremental backups.
> > Also the following flow
> > 1. Run a VM using imageA
> > 2. write to disk
> > 3. Stop VM
> > 4. copy imageA to repoB
> > 5. Run a VM using imageA again
> > 6. Write to disk
> > 7. Stop VM
> > 8. Copy imageA again basing it of imageA_copy1 on repoB creating a
> > diff on repo diff without snapshotting the original image.
> > 
> > > 
> > > >
> > > > return the Id of the new image. In case of copying an immutable
> > > > image the ID will be identical to the original image as they
> > > > contain the same data. However the user should not assume that
> > > > and
> > > > always use the value returned from the method.
> > > >
> > > > removeImage(repositoryId, imageId, options={}):
> > > > repositoryId - The ID of a connected repo where the image to
> > > > delete
> > > > resides
> > > > imageId - The id of the image you wish to delete.
> > > >
> > > >
> > > > ----
> > > > getImageStatus(repositoryId, imageId)
> > > > repositoryId - The ID of a connected repo where the image to
> > > > check
> > > > resides
> > > > imageId - The id of the image you wish to check.
> > > >
> > > > All operations return once the operations has been committed to
> > > > disk NOT when the operation actually completes.
> > > > This is done so that:
> > > > - operation come to a stable state as quickly as possible.
> > > > - In case where there is an SDM, only small portion of the
> > > > operation actually needs to be performed on the SDM host.
> > > > - No matter how many times the operation fails and on how many
> > > > hosts, you can always resume the operation and choose when to
> > > > do
> > > > it.
> > > > - You can stop an operation at any time and remove the
> > > > resulting
> > > > object making a distinction between "stop because the host is
> > > > overloaded" to "I don't want that image"
> > > >
> > > > This means that after calling any operation that creates a new
> > > > image the user must then call getImageStatus() to check what is
> > > > the status of the image.
> > > > The status of the image can be either optimized, degraded, or
> > > > broken.
> > > > "Optimized" means that the image is available and you can run
> > > > VMs
> > > > of it.
> > > > "Degraded" means that the image is available and will run VMs
> > > > but
> > > > it might be a better way VDSM can represent the underlying
> > > > data.
> > > 
> > > What does the "represent" mean here?
> > Anything, but mostly image formate RAW\QCOW2 when performance
> > strategy has been selected.
> > > > "Broken" means that the image can't be used at the moment,
> > > > probably
> > > > because not all the data has been set up on the volume.
> > > >
> > > > Apart from that VDSM will also return the last persisted status
> > > > information which will conatin
> > > > hostID - the last host to try and optimize of fix the image
> > > Any host can optimize the image? No need to be SDM?
> > On anything but lvm based block domains there will not even be an
> > SDM.
> > On SDM based domains we will try as hard as we can to have as many
> > operations executable on any host.
> > > 
> > > > stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> > > > percent_complete - -1 or 0-100, the last persisted completion
> > > > percentage of the aforementioned stage. -1 means that no
> > > > progress
> > > > is available for that operation.
> > > > last_error - This will only be filled if the operation failed
> > > > because of something other then IO or a VDSM crash for obvious
> > > > reasons.
> > > >               It will usually be set if the task was manually
> > > >               stopped
> > > >
> > > > The user can either be satisfied with that information or as
> > > > the
> > > > host specified in host ID if it is still working on that image
> > > > by
> > > > checking it's running tasks.
> > > 
> > > So we need a function to know what tasks are running on the image
> > getImageStatus()
> > > >
> > > > checkStorageRepository(self, repositoryId, options={}):
> > > > A method to go over a storage repository and scan for any
> > > > existing
> > > > problems. This includes degraded\broken images and deleted
> > > > images
> > > > that have no yet been physically deleted\merged.
> > > > It returns a list of Fix objects.
> > > > Fix objects come in 4 types:
> > > > clean - cleans data, run them to get more space.
> > > > optimize - run them to optimize a degraded image
> > > > merge - Merges two images together. Doing this sometimes
> > > >          makes more images ready optimizing or cleaning.
> > > >          The reason it is different from optimize is that
> > > >          unmerged images are considered optimized.
> > > > mend - mends a broken image
> > > >
> > > > The user can read these types and prioritize fixes. Fixes also
> > > > contain opaque FIX data and they should be sent as received to
> > > > fixStorageRepository(self, repositoryId, fix, options={}):
> > > >
> > > > That will start a fix operation.
> > > >
> > > >
> > > > All major operations automatically start the appropriate "Fix"
> > > > to
> > > > bring the created object to an optimize\degraded state (the one
> > > > that is quicker) unless one of the options is
> > > > AutoFix=False. This is only useful for repos that might not be
> > > > able
> > > > to create volumes on all hosts (SDM) but would like to have the
> > > > actual IO distributed in the cluster.
> > > >
> > > > Other common options is the strategy option:
> > > > It has currently 2 possible values
> > > > space and performance - In case VDSM has 2 ways of completing
> > > > the
> > > > same operation it will tell it to value one over the other. For
> > > > example, whether to copy all the data or just create a qcow
> > > > based
> > > > of a snapshot.
> > > > The default is space.
> > > >
> > > > You might have also noticed that it is never explicitly
> > > > specified
> > > > where to look for existing images. This is done purposefully,
> > > > VDSM
> > > > will always look in all connected repositories for existing
> > > > objects.
> > > > For very large setups this might be problematic. To mitigate
> > > > the
> > > > problem you have these options:
> > > > participatingRepositories=[repoId, ...] which tell VDSM to
> > > > narrow
> > > > the search to just these repositories
> > > > and
> > > > imageHints={imgId: repoId} which will force VDSM to look for
> > > > those
> > > > image ID just in those repositories and fail if it doesn't find
> > > > them there.
> > > > _______________________________________________
> > > > vdsm-devel mailing list
> > > > vdsm-devel@lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > > 
> > > 
> > > --
> > > ---
> > > 舒明 Shu Ming
> > > Open Virtualization Engineerning; CSTL, IBM Corp.
> > > Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com
> > > or
> > > shuming@linux.vnet.ibm.com
> > > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> > > District, Beijing 100193, PRC
> > > 
> > > 
> > > 
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> --
> Adam Litke <agl@us.ibm.com>
> IBM Linux Technology Center
> 
>

----- Original Message -----

From: "Adam Litke" <agl@us.ibm.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org> Sent: Monday, December 10, 2012 4:47:46 PM Subject: Re: [vdsm] RFC: New Storage API

On Mon, Dec 10, 2012 at 03:36:23PM -0500, Saggi Mizrahi wrote:

...

...

Statements like this make me start to worry about your userData concept. It's a sign of a bad API if the user needs to invent a custom metadata scheme for itself. This reminds me of the abomination that is the 'custom' property in the vm definition today. In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.

userData being a "void*" is quite common and I don't understand why you would thing it's a sign of a bad API. Further more, giving the user choice about how to represent it's own metadata and what fields it want to keep seems reasonable to me. Especially given the fact that VDSM never reads it.

The reason we are pulling away from the current system of VDSM understanding the extra data is that it makes that data tied to VDSMs on disk format. VDSM on disk format has to be very stable because of clusters with multiple VDSM versions. Further more, since this is actually manager data it has to be tied to the manager backward compatibility lifetime as well. Having it be opaque to VDSM ties it to only one, simpler, support lifetime instead of two.

I guess you are implying that it will make it problematic for multiple users to read userData left by another user because the formats might not be compatible. The solution is that all parties interested in using VDSM storage agree on format, and common fields, and supportability, and all the other things that choosing a supporting *something* entails. This is, however, out of the scope of VDSM. When the time comes I think how the userData blob is actually parsed and what fields it keeps should be discussed on ovirt-devel or engine-devel.

The crux of the issue is that VDSM manages only what it cares about and the user can't modify directly. This is done because everything we expose we commit to. If you want any information persisted like: - Human readable name (in whatever encoding) - Is this a template or a snapshot - What user owns this image

You can just put it in the userData. VDSM is not going to impose what encoding you use. It's not going to decide if you represent your users as IDs or names or ldap queries or Public Keys. It's not going to decide if you have explicit templates or not. It's not going to decide if you care what is the logical image chain. It's not going to decide anything that is out of it's scope. No format is future proof, no selection of fields will be good for any situation. I'd much rather it be someone else's problem when any of them need to be changed. They have currently been VDSMs problem and it has been hell to maintain.

In general, I actually agree with most of this. What I want to avoid is pushing things that should actually be a part of the API into this userData blob. We do want to keep the API as simple as possible to give vdsm flexibility. If, over time, we find that users are always using userData to work around something missing in the API, this could be a really good sign that the API needs extension. I was actually contemplating about this for quite a while. If while you create an image the reply is lost or, VDSM is unable to know if the operation was committed or not, the user will have no way of knowing what thew new image ID is. To solve this it is recommended that the manager puts some sort of task related information in the user data. If the operation ever finishes in an an ambiguous state the user just reads the userData from any images it doesn't know or is unsure about their state.

This is a flow that every client will have to have. So why not just add that to the API? Because I don't want to impose how this "information" gets generated, what is the content of that data or how unique it has to be. Since VDSM doesn't use it for anything, I don't feel like I need to figure this out. I am all for simplicity, but simplicity is kind of an abstract concept. Having it be a blob is in some aspects the simplest thing you can do. Just saying that "I have a field, put whatever in it" is simple to convey but does requires more work on the user's side to figure out what to do with it. All that being said, I do think that the format, fields and how to use them should be defined so different users can communicate and synchronize. It's also important that you don't reinvent the wheel for every flow in every client. I'm just saying that it's not in the scope of VDSM. It should be done as a standard that all users of VDSM agree too conform to. It's the same way that a file-system doesn't check that every *.PDF file is a proper PDF file. PDF clients agree on how to read\save files and users promise to mark them with that suffix. Think of what happened if the FS did check that kind of thing. PDF features will be bound to the kernel release cycle so it doesn't refuse to save\load a file just because your PDF client is newer or older than the kernel. You will have to have code to validate PDF files in the kernel. That means more code which, in turn, means more bugs.

-- Adam Litke <agl@us.ibm.com> IBM Linux Technology Center

----- Original Message -----
> From: "Shu Ming" <shuming@linux.vnet.ibm.com>
> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> Cc: "Adam Litke" <agl@us.ibm.com>, "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development"
> <vdsm-devel@lists.fedorahosted.org>
> Sent: Monday, December 10, 2012 10:33:23 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> 2012-12-11 4:36, Saggi Mizrahi:
> >
> > ----- Original Message -----
> >> From: "Adam Litke" <agl@us.ibm.com>
> >> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> >> Cc: "Shu Ming" <shuming@linux.vnet.ibm.com>, "engine-devel"
> >> <engine-devel@ovirt.org>, "VDSM Project Development"
> >> <vdsm-devel@lists.fedorahosted.org>
> >> Sent: Monday, December 10, 2012 1:39:51 PM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:
> >>>
> >>> ----- Original Message -----
> >>>> From: "Shu Ming" <shuming@linux.vnet.ibm.com>
> >>>> To: "Saggi Mizrahi" <smizrahi@redhat.com>
> >>>> Cc: "VDSM Project Development"
> >>>> <vdsm-devel@lists.fedorahosted.org>, "engine-devel"
> >>>> <engine-devel@ovirt.org>
> >>>> Sent: Thursday, December 6, 2012 11:02:02 AM
> >>>> Subject: Re: [vdsm] RFC: New Storage API
> >>>>
> >>>> Saggi,
> >>>>
> >>>> Thanks for sharing your thought and I get some comments below.
> >>>>
> >>>>
> >>>> Saggi Mizrahi:
> >>>>> I've been throwing a lot of bits out about the new storage API
> >>>>> and
> >>>>> I think it's time to talk a bit.
> >>>>> I will purposefully try and keep implementation details away
> >>>>> and
> >>>>> concentrate about how the API looks and how you use it.
> >>>>>
> >>>>> First major change is in terminology, there is no long a
> >>>>> storage
> >>>>> domain but a storage repository.
> >>>>> This change is done because so many things are already called
> >>>>> domain in the system and this will make things less confusing
> >>>>> for
> >>>>> new-commers with a libvirt background.
> >>>>>
> >>>>> One other changes is that repositories no longer have a UUID.
> >>>>> The UUID was only used in the pool members manifest and is no
> >>>>> longer needed.
> >>>>>
> >>>>>
> >>>>> connectStorageRepository(repoId, repoFormat,
> >>>>> connectionParameters={}):
> >>>>> repoId - is a transient name that will be used to refer to the
> >>>>> connected domain, it is not persisted and doesn't have to be
> >>>>> the
> >>>>> same across the cluster.
> >>>>> repoFormat - Similar to what used to be type (eg. localfs-1.0,
> >>>>> nfs-3.4, clvm-1.2).
> >>>>> connectionParameters - This is format specific and will used to
> >>>>> tell VDSM how to connect to the repo.
> >>>>
> >>>> Where does repoID come from? I think repoID doesn't exist before
> >>>> connectStorageRepository() return.  Isn't repoID a return value
> >>>> of
> >>>> connectStorageRepository()?
> >>> No, repoIDs are no longer part of the domain, they are just a
> >>> transient handle.
> >>> The user can put whatever it wants there as long as it isn't
> >>> already taken by another currently connected domain.
> >>>>> disconnectStorageRepository(self, repoId)
> >>>>>
> >>>>>
> >>>>> In the new API there are only images, some images are mutable
> >>>>> and
> >>>>> some are not.
> >>>>> mutable images are also called VirtualDisks
> >>>>> immutable images are also called Snapshots
> >>>>>
> >>>>> There are no explicit templates, you can create as many images
> >>>>> as
> >>>>> you want from any snapshot.
> >>>>>
> >>>>> There are 4 major image operations:
> >>>>>
> >>>>>
> >>>>> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >>>>>                     userData={}, options={}):
> >>>>>
> >>>>> targetRepoId - ID of a connected repo where the disk will be
> >>>>> created
> >>>>> size - The size of the image you wish to create
> >>>>> baseSnapshotId - the ID of the snapshot you want the base the
> >>>>> new
> >>>>> virtual disk on
> >>>>> userData - optional data that will be attached to the new VD,
> >>>>> could
> >>>>> be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>>>
> >>>>> returns the id of the new VD
> >>>> I think we will also need a function to check if a a VirtualDisk
> >>>> is
> >>>> based on a specific snapshot.
> >>>> Like: isSnapshotOf(virtualDiskId, baseSnapshotID):
> >>> No, the design is that volume dependencies are an implementation
> >>> detail.
> >>> There is no reason for you to know that an image is physically a
> >>> snapshot of another.
> >>> Logical snapshots, template information, and any other
> >>> information
> >>> can be set by the user by using the userData field available for
> >>> every image.
> >> Statements like this make me start to worry about your userData
> >> concept.  It's a
> >> sign of a bad API if the user needs to invent a custom metadata
> >> scheme for
> >> itself.  This reminds me of the abomination that is the 'custom'
> >> property in the
> >> vm definition today.
> > In one sentence: If VDSM doesn't care about it, VDSM doesn't manage
> > it.
> >
> > userData being a "void*" is quite common and I don't understand why
> > you would thing it's a sign of a bad API.
> > Further more, giving the user choice about how to represent it's
> > own metadata and what fields it want to keep seems reasonable to
> > me.
> > Especially given the fact that VDSM never reads it.
> >
> > The reason we are pulling away from the current system of VDSM
> > understanding the extra data is that it makes that data tied to
> > VDSMs on disk format.
> > VDSM on disk format has to be very stable because of clusters with
> > multiple VDSM versions.
> > Further more, since this is actually manager data it has to be tied
> > to the manager backward compatibility lifetime as well.
> > Having it be opaque to VDSM ties it to only one, simpler, support
> > lifetime instead of two.
> 
> Making userData being opaque gives flexibilities to the management
> applications.  To me, opaque userDaa can have two types at least. The
> first is the userData for runtime only.  The second is the userData
> expected to be persisted into the metadata disk.  For the first type,
> the management applications can store their own data structures like
> temporary task states, VDSM query caches &etc. After the VDSM host is
> fenced, the userData goes away. For the second type, different
> management applications can have different formats of userData and
> persisting it to the VDSM metadata disk will corrupt each other in
> very
> high possibility.  You may argue that, all the management
> applications
> should reach a agreement before they use it as the second type.  But
> in
> practice, it is pretty hard to maintain.
We are talking only about the second type.
And yes, if not all clients agree about the format and don't check that they are modifying an image with someone else's userdata you will have problems.
This is the same problem you have if any program starts randomly messing with any other programs persistent data.
Users will have to agree on a format anyway if the want to use the same storage domains because they'll be sharing the same data anyway.
The thing is that if you are making anything other then a manager you should develop against the engine API.
If you are writing your own manager and are also sharing the host with another manager you should really not mess with other manager's repos.
That being said, agreeing on a set of fields and format to be able to have some sort of interoperability between managers is a good idea IMO.
> 
> 
> >
> > I guess you are implying that it will make it problematic for
> > multiple users to read userData left by another user because the
> > formats might not be compatible.
> > The solution is that all parties interested in using VDSM storage
> > agree on format, and common fields, and supportability, and all
> > the other things that choosing a supporting *something* entails.
> > This is, however, out of the scope of VDSM. When the time comes I
> > think how the userData blob is actually parsed and what fields it
> > keeps should be discussed on ovirt-devel or engine-devel.
> >
> > The crux of the issue is that VDSM manages only what it cares about
> > and the user can't modify directly.
> > This is done because everything we expose we commit to.
> > If you want any information persisted like:
> > - Human readable name (in whatever encoding)
> > - Is this a template or a snapshot
> > - What user owns this image
> >
> > You can just put it in the userData.
> > VDSM is not going to impose what encoding you use.
> > It's not going to decide if you represent your users as IDs or
> > names or ldap queries or Public Keys.
> > It's not going to decide if you have explicit templates or not.
> > It's not going to decide if you care what is the logical image
> > chain.
> > It's not going to decide anything that is out of it's scope.
> > No format is future proof, no selection of fields will be good for
> > any situation.
> > I'd much rather it be someone else's problem when any of them need
> > to be changed.
> > They have currently been VDSMs problem and it has been hell to
> > maintain.
> >
> >>>>> createSnapshot(targetRepoId, baseVirtualDiskId,
> >>>>>                  userData={}, options={}):
> >>>>> targetRepoId - The ID of a connected repo where the new
> >>>>> sanpshot
> >>>>> will be created and the original image exists as well.
> >>>>> size - The size of the image you wish to create
> >>>>> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you
> >>>>> want
> >>>>> to snapshot
> >>>>> userData - optional data that will be attached to the new
> >>>>> Snapshot,
> >>>>> could be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>>>
> >>>>> returns the id of the new Snapshot
> >>>>>
> >>>>> copyImage(targetRepoId, imageId, baseImageId=None, userData={},
> >>>>> options={})
> >>>>> targetRepoId - The ID of a connected repo where the new image
> >>>>> will
> >>>>> be created
> >>>>> imageId - The image you wish to copy
> >>>>> baseImageId - if specified, the new image will contain only the
> >>>>> diff between image and Id.
> >>>>>                 If None the new image will contain all the bits
> >>>>>                 of
> >>>>>                 image Id. This can be used to copy partial
> >>>>>                 parts
> >>>>>                 of
> >>>>>                 images for export.
> >>>>> userData - optional data that will be attached to the new
> >>>>> image,
> >>>>> could be anything that the user desires.
> >>>>> options - options to modify VDSMs default behavior
> >>>> Does this function mean that we can copy the image from one
> >>>> repository
> >>>> to another repository? Does it cover the semantics of storage
> >>>> migration,
> >>>> storage backup, storage incremental backup?
> >>> Yes, the main purpose is copying to another repo. and you can
> >>> even
> >>> do incremental backups.
> >>> Also the following flow
> >>> 1. Run a VM using imageA
> >>> 2. write to disk
> >>> 3. Stop VM
> >>> 4. copy imageA to repoB
> >>> 5. Run a VM using imageA again
> >>> 6. Write to disk
> >>> 7. Stop VM
> >>> 8. Copy imageA again basing it of imageA_copy1 on repoB creating
> >>> a
> >>> diff on repo diff without snapshotting the original image.
> >>>
> >>>>> return the Id of the new image. In case of copying an immutable
> >>>>> image the ID will be identical to the original image as they
> >>>>> contain the same data. However the user should not assume that
> >>>>> and
> >>>>> always use the value returned from the method.
> >>>>>
> >>>>> removeImage(repositoryId, imageId, options={}):
> >>>>> repositoryId - The ID of a connected repo where the image to
> >>>>> delete
> >>>>> resides
> >>>>> imageId - The id of the image you wish to delete.
> >>>>>
> >>>>>
> >>>>> ----
> >>>>> getImageStatus(repositoryId, imageId)
> >>>>> repositoryId - The ID of a connected repo where the image to
> >>>>> check
> >>>>> resides
> >>>>> imageId - The id of the image you wish to check.
> >>>>>
> >>>>> All operations return once the operations has been committed to
> >>>>> disk NOT when the operation actually completes.
> >>>>> This is done so that:
> >>>>> - operation come to a stable state as quickly as possible.
> >>>>> - In case where there is an SDM, only small portion of the
> >>>>> operation actually needs to be performed on the SDM host.
> >>>>> - No matter how many times the operation fails and on how many
> >>>>> hosts, you can always resume the operation and choose when to
> >>>>> do
> >>>>> it.
> >>>>> - You can stop an operation at any time and remove the
> >>>>> resulting
> >>>>> object making a distinction between "stop because the host is
> >>>>> overloaded" to "I don't want that image"
> >>>>>
> >>>>> This means that after calling any operation that creates a new
> >>>>> image the user must then call getImageStatus() to check what is
> >>>>> the status of the image.
> >>>>> The status of the image can be either optimized, degraded, or
> >>>>> broken.
> >>>>> "Optimized" means that the image is available and you can run
> >>>>> VMs
> >>>>> of it.
> >>>>> "Degraded" means that the image is available and will run VMs
> >>>>> but
> >>>>> it might be a better way VDSM can represent the underlying
> >>>>> data.
> >>>> What does the "represent" mean here?
> >>> Anything, but mostly image formate RAW\QCOW2 when performance
> >>> strategy has been selected.
> >>>>> "Broken" means that the image can't be used at the moment,
> >>>>> probably
> >>>>> because not all the data has been set up on the volume.
> >>>>>
> >>>>> Apart from that VDSM will also return the last persisted status
> >>>>> information which will conatin
> >>>>> hostID - the last host to try and optimize of fix the image
> >>>> Any host can optimize the image? No need to be SDM?
> >>> On anything but lvm based block domains there will not even be an
> >>> SDM.
> >>> On SDM based domains we will try as hard as we can to have as
> >>> many
> >>> operations executable on any host.
> >>>>> stage - X/Y (eg. 1/10) the last persisted stage of the fix.
> >>>>> percent_complete - -1 or 0-100, the last persisted completion
> >>>>> percentage of the aforementioned stage. -1 means that no
> >>>>> progress
> >>>>> is available for that operation.
> >>>>> last_error - This will only be filled if the operation failed
> >>>>> because of something other then IO or a VDSM crash for obvious
> >>>>> reasons.
> >>>>>                It will usually be set if the task was manually
> >>>>>                stopped
> >>>>>
> >>>>> The user can either be satisfied with that information or as
> >>>>> the
> >>>>> host specified in host ID if it is still working on that image
> >>>>> by
> >>>>> checking it's running tasks.
> >>>> So we need a function to know what tasks are running on the
> >>>> image
> >>> getImageStatus()
> >>>>> checkStorageRepository(self, repositoryId, options={}):
> >>>>> A method to go over a storage repository and scan for any
> >>>>> existing
> >>>>> problems. This includes degraded\broken images and deleted
> >>>>> images
> >>>>> that have no yet been physically deleted\merged.
> >>>>> It returns a list of Fix objects.
> >>>>> Fix objects come in 4 types:
> >>>>> clean - cleans data, run them to get more space.
> >>>>> optimize - run them to optimize a degraded image
> >>>>> merge - Merges two images together. Doing this sometimes
> >>>>>           makes more images ready optimizing or cleaning.
> >>>>>           The reason it is different from optimize is that
> >>>>>           unmerged images are considered optimized.
> >>>>> mend - mends a broken image
> >>>>>
> >>>>> The user can read these types and prioritize fixes. Fixes also
> >>>>> contain opaque FIX data and they should be sent as received to
> >>>>> fixStorageRepository(self, repositoryId, fix, options={}):
> >>>>>
> >>>>> That will start a fix operation.
> >>>>>
> >>>>>
> >>>>> All major operations automatically start the appropriate "Fix"
> >>>>> to
> >>>>> bring the created object to an optimize\degraded state (the one
> >>>>> that is quicker) unless one of the options is
> >>>>> AutoFix=False. This is only useful for repos that might not be
> >>>>> able
> >>>>> to create volumes on all hosts (SDM) but would like to have the
> >>>>> actual IO distributed in the cluster.
> >>>>>
> >>>>> Other common options is the strategy option:
> >>>>> It has currently 2 possible values
> >>>>> space and performance - In case VDSM has 2 ways of completing
> >>>>> the
> >>>>> same operation it will tell it to value one over the other. For
> >>>>> example, whether to copy all the data or just create a qcow
> >>>>> based
> >>>>> of a snapshot.
> >>>>> The default is space.
> >>>>>
> >>>>> You might have also noticed that it is never explicitly
> >>>>> specified
> >>>>> where to look for existing images. This is done purposefully,
> >>>>> VDSM
> >>>>> will always look in all connected repositories for existing
> >>>>> objects.
> >>>>> For very large setups this might be problematic. To mitigate
> >>>>> the
> >>>>> problem you have these options:
> >>>>> participatingRepositories=[repoId, ...] which tell VDSM to
> >>>>> narrow
> >>>>> the search to just these repositories
> >>>>> and
> >>>>> imageHints={imgId: repoId} which will force VDSM to look for
> >>>>> those
> >>>>> image ID just in those repositories and fail if it doesn't find
> >>>>> them there.
> >>>>> _______________________________________________
> >>>>> vdsm-devel mailing list
> >>>>> vdsm-devel@lists.fedorahosted.org
> >>>>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>>
> >>>> --
> >>>> ---
> >>>> 舒明 Shu Ming
> >>>> Open Virtualization Engineerning; CSTL, IBM Corp.
> >>>> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com
> >>>> or
> >>>> shuming@linux.vnet.ibm.com
> >>>> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> >>>> District, Beijing 100193, PRC
> >>>>
> >>>>
> >>>>
> >>> _______________________________________________
> >>> vdsm-devel mailing list
> >>> vdsm-devel@lists.fedorahosted.org
> >>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >> --
> >> Adam Litke <agl@us.ibm.com>
> >> IBM Linux Technology Center
> >>
> >>
> 
> 
> --
> ---
> 舒明 Shu Ming
> Open Virtualization Engineerning; CSTL, IBM Corp.
> Tel: 86-10-82451626  Tieline: 9051626 E-mail: shuming@cn.ibm.com or
> shuming@linux.vnet.ibm.com
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
> District, Beijing 100193, PRC
> 
> 
>

----- Original Message -----

From: "Itamar Heim" <iheim@redhat.com> To: "Saggi Mizrahi" <smizrahi@redhat.com> Cc: "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org>, "engine-devel" <engine-devel@ovirt.org> Sent: Monday, January 14, 2013 6:18:13 AM Subject: Re: [vdsm] RFC: New Storage API

On 12/04/2012 11:52 PM, Saggi Mizrahi wrote:

...

I've been throwing a lot of bits out about the new storage API and I think it's time to talk a bit. I will purposefully try and keep implementation details away and concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a storage domain but a storage repository. This change is done because so many things are already called domain in the system and this will make things less confusing for new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID. The UUID was only used in the pool members manifest and is no longer needed.

connectStorageRepository(repoId, repoFormat, connectionParameters={}): repoId - is a transient name that will be used to refer to the connected domain, it is not persisted and doesn't have to be the same across the cluster. repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, clvm-1.2). connectionParameters - This is format specific and will used to tell VDSM how to connect to the repo.

disconnectStorageRepository(self, repoId):

In the new API there are only images, some images are mutable and some are not. mutable images are also called VirtualDisks immutable images are also called Snapshots

There are no explicit templates, you can create as many images as you want from any snapshot.

There are 4 major image operations:

createVirtualDisk(targetRepoId, size, baseSnapshotId=None, userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be created size - The size of the image you wish to create baseSnapshotId - the ID of the snapshot you want the base the new virtual disk on userData - optional data that will be attached to the new VD, could be anything that the user desires. options - options to modify VDSMs default behavior

returns the id of the new VD

createSnapshot(targetRepoId, baseVirtualDiskId, userData={}, options={}): targetRepoId - The ID of a connected repo where the new sanpshot will be created and the original image exists as well. size - The size of the image you wish to create baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot userData - optional data that will be attached to the new Snapshot, could be anything that the user desires. options - options to modify VDSMs default behavior

returns the id of the new Snapshot

copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={}) targetRepoId - The ID of a connected repo where the new image will be created imageId - The image you wish to copy baseImageId - if specified, the new image will contain only the diff between image and Id. If None the new image will contain all the bits of image Id. This can be used to copy partial parts of images for export. userData - optional data that will be attached to the new image, could be anything that the user desires. options - options to modify VDSMs default behavior

return the Id of the new image. In case of copying an immutable image the ID will be identical to the original image as they contain the same data. However the user should not assume that and always use the value returned from the method.

removeImage(repositoryId, imageId, options={}): repositoryId - The ID of a connected repo where the image to delete resides imageId - The id of the image you wish to delete.

---- getImageStatus(repositoryId, imageId) repositoryId - The ID of a connected repo where the image to check resides imageId - The id of the image you wish to check.

All operations return once the operations has been committed to disk NOT when the operation actually completes. This is done so that: - operation come to a stable state as quickly as possible. - In case where there is an SDM, only small portion of the operation actually needs to be performed on the SDM host. - No matter how many times the operation fails and on how many hosts, you can always resume the operation and choose when to do it. - You can stop an operation at any time and remove the resulting object making a distinction between "stop because the host is overloaded" to "I don't want that image"

This means that after calling any operation that creates a new image the user must then call getImageStatus() to check what is the status of the image. The status of the image can be either optimized, degraded, or broken. "Optimized" means that the image is available and you can run VMs of it. "Degraded" means that the image is available and will run VMs but it might be a better way VDSM can represent the underlying data. "Broken" means that the image can't be used at the moment, probably because not all the data has been set up on the volume.

Apart from that VDSM will also return the last persisted status information which will conatin hostID - the last host to try and optimize of fix the image stage - X/Y (eg. 1/10) the last persisted stage of the fix. percent_complete - -1 or 0-100, the last persisted completion percentage of the aforementioned stage. -1 means that no progress is available for that operation. last_error - This will only be filled if the operation failed because of something other then IO or a VDSM crash for obvious reasons. It will usually be set if the task was manually stopped

The user can either be satisfied with that information or as the host specified in host ID if it is still working on that image by checking it's running tasks.

checkStorageRepository(self, repositoryId, options={}): A method to go over a storage repository and scan for any existing problems. This includes degraded\broken images and deleted images that have no yet been physically deleted\merged. It returns a list of Fix objects. Fix objects come in 4 types: clean - cleans data, run them to get more space. optimize - run them to optimize a degraded image merge - Merges two images together. Doing this sometimes makes more images ready optimizing or cleaning. The reason it is different from optimize is that unmerged images are considered optimized. mend - mends a broken image

The user can read these types and prioritize fixes. Fixes also contain opaque FIX data and they should be sent as received to fixStorageRepository(self, repositoryId, fix, options={}):

That will start a fix operation.

All major operations automatically start the appropriate "Fix" to bring the created object to an optimize\degraded state (the one that is quicker) unless one of the options is AutoFix=False. This is only useful for repos that might not be able to create volumes on all hosts (SDM) but would like to have the actual IO distributed in the cluster.

Other common options is the strategy option: It has currently 2 possible values space and performance - In case VDSM has 2 ways of completing the same operation it will tell it to value one over the other. For example, whether to copy all the data or just create a qcow based of a snapshot. The default is space.

You might have also noticed that it is never explicitly specified where to look for existing images. This is done purposefully, VDSM will always look in all connected repositories for existing objects. For very large setups this might be problematic. To mitigate the problem you have these options: participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to just these repositories and imageHints={imgId: repoId} which will force VDSM to look for those image ID just in those repositories and fail if it doesn't find them there. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

you are using VirtualDisk and Snapshot for mutable and immutable. the terminology of "image" for immutable and "volume" for mutable seems to be the one used by ec2/openstack/etc. - any thoughts on using similar terminology? (though i think they also differ in forcing all images to be in an image repo, and all volumes in a volume repo, while we allow to mix them in same repo).

We have volumes internally though, I wanted to call them slabs but Ayal and Edu feel strongly about calling them volumes and the current naming system in general.

2013-1-15 5:34, Ayal Baron:

image and volume are overused everywhere and it would be extremely confusing to have multiple meanings to the same terms in the same system (we have image today which means virtual disk and volume which means a part of a virtual disk). Personally I don't like the distinction between image and volume done in ec2/openstack/etc seeing as they're treated as different types of entities there while the only real difference is mutability (images are read-only, volumes are read-write). To move to the industry terminology we would need to first change all references we have today to image and volume in the system (I would say also in ovirt-engine side) to align with the new meaning. Despite my personal dislike of the terms, I definitely see the value in converging on the same terminology as the rest of the industry but to do so would be an arduous task which is out of scope of this discussion imo (patches welcome though ;)

Another distinction between Openstack and oVirt is how the Nova/ovirt-engine look upon storage systems. In Openstack, a stand alone storage service(Cinder) exports the raw storage block device to Nova. On the other hand, in oVirt, storage system is highly bounded with the cluster scheduling system which integrates storage sub-system, VM dispatching sub-system, ISO image sub systems. This combination make all of the sub-system integrated in a whole which is easy to deploy, but it make the sub-system more opaque and not harder to reuse and maintain. This new storage API proposal give us an opportunity to distinct these sub-systems as new components which export better, loose-coupling APIs to VDSM.

...

_______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel

-- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail:shuming@cn.ibm.com orshuming@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

----- Original Message -----

From: "Adam Litke" <agl@us.ibm.com> To: "Shu Ming" <shuming@linux.vnet.ibm.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "VDSM Project Development" <vdsm-devel@lists.fedorahosted.org> Sent: Tuesday, January 22, 2013 2:20:19 PM Subject: Re: [vdsm] [Engine-devel] RFC: New Storage API

On Tue, Jan 22, 2013 at 11:36:57PM +0800, Shu Ming wrote:

...

2013-1-15 5:34, Ayal Baron:

...

image and volume are overused everywhere and it would be extremely confusing to have multiple meanings to the same terms in the same system (we have image today which means virtual disk and volume which means a part of a virtual disk). Personally I don't like the distinction between image and volume done in ec2/openstack/etc seeing as they're treated as different types of entities there while the only real difference is mutability (images are read-only, volumes are read-write). To move to the industry terminology we would need to first change all references we have today to image and volume in the system (I would say also in ovirt-engine side) to align with the new meaning. Despite my personal dislike of the terms, I definitely see the value in converging on the same terminology as the rest of the industry but to do so would be an arduous task which is out of scope of this discussion imo (patches welcome though ;)

Another distinction between Openstack and oVirt is how the Nova/ovirt-engine look upon storage systems. In Openstack, a stand alone storage service(Cinder) exports the raw storage block device to Nova. On the other hand, in oVirt, storage system is highly bounded with the cluster scheduling system which integrates storage sub-system, VM dispatching sub-system, ISO image sub systems. This combination make all of the sub-system integrated in a whole which is easy to deploy, but it make the sub-system more opaque and not harder to reuse and maintain. This new storage API proposal give us an opportunity to distinct these sub-systems as new components which export better, loose-coupling APIs to VDSM.

A very good point and an important goal in my opinion. I'd like to see ovirt-engine become more of a GUI for configuring the storage component (like it does for Gluster) rather than the centralized manager of storage. The clustered storage should be able to take care of itself as long as the peer hosts can negotiate the SDM role.

It would be cool if someone could actually dedicate a non-virtualization host where its only job is to handle SDM operations. Such a host could choose to only deploy the standalone HSM service and not the complete vdsm package.

OpenStack and oVirt have different architectures and goals. Even though they are both marketed as IaaS solutions they are designed for different purposes. OpenStack is designed around the idea of simplifying the *development* and *integration* of IaaS subsystems through standardization of interfaces. If you design a system that requires access to some type of infrastructural resource you can develop against the OpenStack API for that specific resource and you can consume different underlying implementations of the subsystem. Alternatively if you are creating a new subsystem implementations one of your exposed APIs can be the appropriate OpenStack API. In short, they are a group of loosely coupled services meant to be used replicated and distributed in a cluster. Everyone can create they own implementations of the APIs. oVirt is designed around the idea of simplifying the *management* of said infrastructure. The ovirt-engine is the cluster manager and VDSM is the host-manager. Every host in the cluster has a host manager installed on it (VDSM) and some (currently only 1) might have the cluster-manager (ovirt-engine) and they are the effective brain. oVirt ideally only has managing entities. VDSM APIs delegate to other subsystems tasks that are in it's scope, the subsystems have their own APIs. For VMs you have libvirt, for networking you have the linux management tools and maybe netcf for policy we now have MOM. For iscsi we have iscsiadm, etc. The only odd one out is the image provisioning subsystem which I will get in to, don't worry. This means, if you didn't already gather, that no host managed by ovirt can exist without either VDSM or the ovirt-engine living on it. That being said, I am a huge proponent of making all subsystems optional. Meaning you can have VDSM that doesn't have the libvirt or networking glue bits and just has storage, and gluster. To put it simply, no host without a *managing* entity on it. But, as you all have pointed out, VDSM is redundant. There is no reason why the engine can't just directly ask libvirt to do things. There is no reason why we can't make a general iscsi management API and expose it on it's own, independent from other services. VDSM is frankensteinesque abomination of misplaced BL and pass-through APIs. This is why everyone are having a hard time figuring out what to do with it. I have people on one side bothering me about putting more *management* logic in. On the other side people are calling for standardized APIs which can't exist with engine specific management logic. To sum up this long rant, if you want to split up the bits that make VDSM in to independent subsystems just don't work on VDSM because VDSM (excluding the storage subsystem) is just glue anyway. Better just write another OpenStack back-end for something. As for the storage core. It is the only thing in VDSM that is not glue and actually needs to sit on the host. The new storage code I'm writing is actually developed in the hope that it will be independent from the rest of VDSM and VDSM will just have glue, like everything else.

-- Adam Litke <agl@us.ibm.com> IBM Linux Technology Center

_______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel