From dfediuck at redhat.com  Sun Jun  2 11:48:59 2013
From: dfediuck at redhat.com (Doron Fediuck)
Date: Sun, 2 Jun 2013 07:48:59 -0400 (EDT)
Subject: Qos feature
In-Reply-To: <51A46194.7080409@redhat.com>
References: <2041139267.3543084.1369294470496.JavaMail.root@redhat.com>
	<51A34118.8060204@redhat.com> <51A34756.9050902@redhat.com>
	<51A35265.2060306@redhat.com> <51A38FFF.8060705@redhat.com>
	<51A46194.7080409@redhat.com>
Message-ID: <508145294.17364399.1370173739561.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Michael Pasternak" <mpastern at redhat.com>
> To: "Livnat Peer" <lpeer at redhat.com>
> Cc: "Ofri Masad" <omasad at redhat.com>, arch at ovirt.org
> Sent: Tuesday, May 28, 2013 10:49:40 AM
> Subject: Re: Qos feature
> 
> On 05/27/2013 07:55 PM, Livnat Peer wrote:
> > On 05/27/2013 03:32 PM, Michael Pasternak wrote:
> >>
> >> Hi Livnat,
> >>
> >> On 05/27/2013 02:45 PM, Livnat Peer wrote:
> >>> On 05/27/2013 02:18 PM, Michael Pasternak wrote:
> >>>> Hi Ofri,
> >>>>
> >>>> On 05/23/2013 10:34 AM, Ofri Masad wrote:
> >>>>> Hi Michael,
> >>>>>
> >>>>> I've drafted the Network QoS feature page.
> >>>>> www.ovirt.org/Features/Design/Network_QoS
> >>>>> I'll appreciate it if you could go over it and see if you have comments
> >>>>> or anything else to add to the API section.
> >>>>>
> >>>>> Thanks,
> >>>>> Ofri
> >>>>>
> >>>>
> >>>> i have few questions/comments regarding api/general design:
> >>>>
> >>>> 1) incoming and outgoing traffic can be shaped independently, will we
> >>>> support
> >>>>    granularity of enabling|disabling qos on inbound vs.outbound
> >>>>    traffic?,
> >>>>    then i'd suggest modelling QOS in api like this:
> >>>>
> >>>> <qos>
> >>>>   <inbound enabled="true|false">
> >>>>     <average>xxx</average>
> >>>>     <peak>yyy</peak>
> >>>>     <burst>zzz</burst>
> >>>>     <floor>qqq</floor>
> >>>>   </inbound>
> >>>>   <outbound enabled="true|false">
> >>>>     <average>xxx</average>
> >>>>     <peak>yyy</peak>
> >>>>     <burst>zzz</burst>
> >>>>   </outbound>
> >>>> </qos>
> >>>
> >>>
> >>> Note that the above requires to support a mode of inbound (or outbound)
> >>> values configured while not enabled,
> >>
> >> no it doesn't, all attributes can have NULLs when disabled.
> >>
> > 
> > Why disable+null values is better than just omitting the
> > inbound/outbound elements?
> 
> basically this design serves my proposal mentioned below,
> so administration can clearly see that no defaults exist,
> and they have to choose policy explicitly.
> 
> but i agree "omitting" is a good way to go as well
> (we actively using it across the api)
> 
> > 
> > I liked the disable/enable idea...
> 
> this is exactly the same idea, just a bit visualized.
> 
> > 
> >>> it also requires a change in the UI.
> >>
> >> api & UI doesn't have to be the same, but i think suggested above makes
> >> sense
> >> for UI as well.
> >>
> > of course, I agree, they don't have to be the same. In this case I think
> > it makes sense.
> > 
> >>> I see how the above suggestion can be useful to users.
> >>
> >> it can be useful for us on a first place, cause it's easily extendible,
> >> i.e it much easier to extend expended type with new elements rather
> >> than inline adding attributes <outbound average='' peak='' burst=''
> >> ...../>
> >> and in addition we can easily add system element's (if not now then in
> >> future)
> >> such as <inbound enabled="true|false" policy='xxx' ...>
> >>
> >>>
> >>>>
> >>>>
> >>>> 2) at design page, i didn't saw you mentioning permission model for the
> >>>> QOS,
> >>>>    who should be able seeing it/manipulating it (note, that in case of
> >>>>    user
> >>>>    that should not be seeing it, in api you shouldn't even show the
> >>>>    state of the
> >>>>    qos e.g active or not, maybe not even worth mentioning <qos> at all)
> >>>>
> >>>
> >>> That's a good point.
> >>> Also note that the above suggests different UI dialogs in the user
> >>> portal and the admin portal for adding vnic.
> >>>
> >>>
> >>>>
> >>>> 3) what about exposing policies?, like making admin being able to apply
> >>>> same policy
> >>>>    on a different devices, rather than doing it manually per device.
> >>>
> >>> I suggest to support having default configuration on the Network entity,
> >>> all Vnics connected to the network inherit the configuration from the
> >>> Network.
> >>> If qos is configured both on the network and on the VNIC the VNIC
> >>> configuration holds.
> >>
> >> usually QoS providers avoid having defaults cause when QoS turned on by
> >> mistake it's simply applies these values silently, while if it has NULLs,
> >> you cannot turn it on by mistake (i.e forcing users to set rules preferred
> >> over default configurations)
> >>
> > 
> > I think the idea wasn't clear enough, a more detailed explanation -
> > 
> > A user can configure the inbound/outbound values on the network entity.
> > This configuration would apply to all VNICs using this network.
> > That approach would help the administrator to configure, in a very
> > simple way, a policy that prevents one VM from taking too much of the
> > network bandwidth.
> 
> i understand, but this is different use-case, i was thinking about
> private policies (i'm sure there are plenty use-cases for it), this
> way we can help by providing policy management, like:
> 
> /api/slapolicies
> 
> <slapolicy id=xxx href=/api/slapolicies>
>   <name>backup_qos_policy</name>
>   <type>network</type>
>   <inbound enabled="true|false">
>     <average>xxx</average>
>     <peak>yyy</peak>
>     <burst>zzz</burst>
>     <floor>qqq</floor>
>   </inbound>
>   <outbound enabled="true|false">
>     <average>xxx</average>
>     <peak>yyy</peak>
>     <burst>zzz</burst>
>   </outbound>
> </slapolicy>
> 
> <slapolicy id=xxx href=/api/slapolicies>
>   <name>user_cpu_policy</name>
>   <type>cpu</type>
>   ...
> </slapolicy>
> 
> <slapolicy id=xxx href=/api/slapolicies>
>   <name>user_memory_policy</name>
>   <type>memory</type>
>   ...
> </slapolicy>
> 
> ...
> 

Michael,
This makes a lot of sense, but in a given context.
For example, each cluster will have its own defaults,
and policy implementations. Since this is still in a
thinking process I'd like to wait with this important
discussion until we have some time to come up with
a more complete policies design.
In the meantime we may want to continue with network
QoS without the REST implementation to prevent a lock
before we can ensure it works well with the whole SLA
design.

> > 
> > 
> > 
> >>>
> >>>>
> >>>>> Change the Virtual Machine > Network Interfaces to support QoS
> >>>>> properties Example of an XML representation of a network interface
> >>>>>
> >>>>> <nic id="7a3cff5e-3cc4-47c2-8388-9adf16341f5e"
> >>>>> ref="/api/vms/cdc0b102-fbfe-444a-b9cb-57d2af94f401/nics/7a3cff5e-3cc4-47c2-8388-9adf16341f5e">
> >>>>>      <link rel="statistics"
> >>>>>      href="/api/vms/082c794b-771f-452f-83c9-b2b5a19c0399/nics/7a3cff5e-3cc4-47c2-8388-9adf16341f5e/statistics"/>
> >>>>>      <name>nic1</name>
> >>>>>      <interface>virtio</interface>
> >>>>>      <mac address="00:1a:4a:16:84:07"/>
> >>>>>      <network id="00000000-0000-0000-0000-000000000009"
> >>>>>      href="/api/networks/00000000-0000-0000-0000-000000000009"/>
> >>>>>      <vm id="cdc0b102-fbfe-444a-b9cb-57d2af94f401"
> >>>>>      href="/api/vms/cdc0b102-fbfe-444a-b9cb-57d2af94f401"/>
> >>>>
> >>>> - 'bandwidth' is not clear enough in my view, i think we should just use
> >>>> 'qos'
> >>>>
> >>>>>      <bandwidth>
> >>>>
> >>>> - we should be having <enabled>true|false</enabled> element here/peer
> >>>> traffic-dist?
> >>>>
> >>>>>        <inbound average='1000' peak='5000' floor='200' burst='1024'/>
> >>>>>        <outbound average='128' peak='256' burst='256'/>
> >>>>>      </bandwidth>
> >>>>> </nic>
> >>>>>
> >>>>>
> >>>>> An API user modifies a network interface with a PUT request
> >>>>>
> >>>>> <nic>
> >>>>>       <name>nic2</name>
> >>>>>      <network id="00000000-0000-0000-0000-000000000010"/>
> >>>>>      <type>e1000</type>
> >>>>>      <bandwidth>
> >>>>>        <inbound average='1000' peak='5000' floor='200' burst='1024'/>
> >>>>>        <outbound average='128' peak='256' burst='256'/>
> >>>>>      </bandwidth>
> >>>>> </nic>
> >>>>
> >>>>
> >>>
> >>
> >>
> > 
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 


From rgolan at redhat.com  Mon Jun  3 07:39:23 2013
From: rgolan at redhat.com (Roy Golan)
Date: Mon, 03 Jun 2013 10:39:23 +0300
Subject: oVirt on IBM POWER (PPC64) - new feature contributors
In-Reply-To: <50EB20226B72D6419356FC320AB62B871896B600@SERV070.corp.eldorado.org.br>
References: <50EB20226B72D6419356FC320AB62B871896B600@SERV070.corp.eldorado.org.br>
Message-ID: <51AC482B.3030807@redhat.com>

On 05/14/2013 09:05 PM, Leonardo Bianconi wrote:
> Dear all
>
> We would like to introduce ourselves: Leonardo Bianconi and Vitor Lima.
>
> We would like to work on the features "Engine support for PPC64" (http://wiki.ovirt.org/Features/Engine_support_for_PPC64) and the "Vdsm for PPC64" (http://wiki.ovirt.org/Features/Vdsm_for_PPC64). This work has already been started by some developers at IBM.
>
> We would be happy to hear opinions and comments.
>
> About libosinfo:
> =============
>
> In the previous discussion about this topic (http://lists.ovirt.org/pipermail/arch/2012-November/000976.html), occurred in November 2012, it was suggested that integrating the libosinfo into the engine would be a better way to handle the differences between the architectures that would be supported in the future.
>
> Is this approach still valid? If so, when will it be available? It seems to be a dependency for oVirt "Engine_support_for_PPC64" feature implementation.
>
> Best regards
>
> Leonardo Bianconi / Vitor Lima
>
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch

hi Leonardo and Vitor.

please join the thread [1] to review the OS info wiki [2] which is related to this work.

[1] http://lists.ovirt.org/pipermail/engine-devel/2013-June/004743.html
[2]http://www.ovirt.org/OS_info



From iheim at redhat.com  Mon Jun  3 08:29:44 2013
From: iheim at redhat.com (Itamar Heim)
Date: Mon, 03 Jun 2013 11:29:44 +0300
Subject: gerrit upgrade
Message-ID: <51AC53F8.7050101@redhat.com>

fyi - I'm planning to upgrade gerrit to 2.5.x next Sunday (june 9th), 
morning IL time.

downtime should be a few minutes.
post the upgrade the github mirroring will stop working, and will be 
re-enabled as a gerrit plugin after gerrit seems stable.

Thanks,
    Itamar


From amuller at redhat.com  Mon Jun  3 09:00:23 2013
From: amuller at redhat.com (Assaf Muller)
Date: Mon, 3 Jun 2013 05:00:23 -0400 (EDT)
Subject: Multiple Gateways 3.3 Feature
In-Reply-To: <51A61F10.3080000@redhat.com>
References: <1038209955.13068617.1369638715648.JavaMail.root@redhat.com>
	<51A61610.7070403@redhat.com> <20130529152610.GA19068@redhat.com>
	<51A61F10.3080000@redhat.com>
Message-ID: <2112193197.16893784.1370250023594.JavaMail.root@redhat.com>

After an initial round of code reviews, Dan and I came up with the following code architecture at the VDSM side:

Additional requirements arose at the architectural level: 
1) Working with the configWriter configurator and the backups system 
2) Working with the upcoming iproute2 configurator: http://gerrit.ovirt.org/#/c/15301/
3) A solution that works properly with both configurators 

As a quick reminder, if the IP configuration is static I need to work with rule and route files, thus I need the cfg file configurator. 
If the IP configuration is DHCP I need to access the ip binary directly and add or remove routing lines and rules. 

The new proposed solution looks like this: 
* iproute2.py wrapper - Wrap the ip binary with our own methods that return named tuples containing the expected output. 
- This is quite a project in itself so I'll only implement the exact methods that I actually need 
- Write a unit test for this module 

* The new SourceRoute class will encapsulate two concepts: RuleSet (A set of rules like those in ip rule) and RoutingTable (Like the main routing table... ip route show table main) 
It's possible to separate the SourceRoute class to two classes but I have a feeling that will cause some duplication of code so right now I intend to implement just one class. 
- The SourceRoute class will be added to netmodels.py 

* The SourceRoute class receives the information it needs (ip, mask, gateway) and a configurator (Either ifcfg files configurator or iproute2 configurator). It has two methods: 
- configure: Call its configurator configureRuleSet and configureRoutingTable methods 
- remove: Call its configurator removeRuleSet and removeRoutingTable methods 
If the IP configuration is static, the cfg files configurator will be used and so the rule and route files will be created 
If the IP configuration is DHCP, the iproute2 configurator will be used and the ip binary will be called directly 

* DHCP hooks: During VDSM installation copy over two Python scripts to /etc/dhcp 
- dhclient-up-hooks, dhclient-down-hooks 
- These are two executables that are called whenever DHCP gets a DHCP response 
- They will check if the interface that was brought up or down is tracked by VDSM, and if so create a new SourceRoute instance and pass a new iproute2 configurator 

* During configNetwork addNetwork and delNetwork: 
If the IP configuration is DHCP, there is nothing to do as the DHCP hooks were already put in place during VDSM installation 
If the IP configuration is static, then get the ifcfg configurator, pass it to a new SourceRoute instance and call configure


From liumbj at linux.vnet.ibm.com  Mon Jun  3 10:28:46 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Mon, 03 Jun 2013 18:28:46 +0800
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <1264434811.14301245.1369841664456.JavaMail.root@redhat.com>
References: <51A46842.1090901@linux.vnet.ibm.com> <51A5B170.50205@redhat.com>
	<51A5BDC0.2020602@linux.vnet.ibm.com>
	<1264434811.14301245.1369841664456.JavaMail.root@redhat.com>
Message-ID: <51AC6FDE.5030007@linux.vnet.ibm.com>

On 05/29/2013 11:34 PM, Doron Fediuck wrote:
> ----- Original Message -----
>> From: "Mei Liu" <liumbj at linux.vnet.ibm.com>
>> To: "Dave Neary" <dneary at redhat.com>
>> Cc: arch at ovirt.org
>> Sent: Wednesday, May 29, 2013 11:35:12 AM
>> Subject: Re: SLA feature for storage I/O bandwidth
>>
>> On 05/29/2013 03:42 PM, Dave Neary wrote:
>>> Hi Mei Liu,
>>>
>>> On 05/28/2013 10:18 AM, Mei Liu wrote:
>>>> I created a drafted wiki page on  design of storage I/O bandwidth SLA in
>>>> the following link:
>>>>
>>>> http://www.ovirt.org/SLA_for_storage_resource .
>>>>
>>>> I will appreciate the efforts if anyone who works on ovirt engine, vdsm
>>>> or mom could give some comments. TIA.
>>> Just out of interest - which version of oVirt are you targeting this
>>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
>>> feature freeze (but we have a release meeting later to discuss that).
>>>
>>> Thanks,
>>> Dave.
>>>
>> Hi Dave,
>> The basic i/o tune functionality for vdsm is almost ready. However,
>> there is nothing written on the Engine side and no policy for automatic
>> tuning is applied yet.
>> I am not sure if the basic functionality can target 3.3.
>>
>>
>> Best regards,
>> Mei Liu
>>
> Hi Mey Liu,
> I'm still going over the wiki, but a few things we need to consider;
> First of all QoS for storage I/O bandwidth is a part of a larger SLA
> policy which may include network QoS, CPU and memory QoS and the quota
> we implement today.
>
> So first of all we need to make sure your design does not conflict
> the other QoS parts, which is what I'm looking into now.
>
> Additionally, using the quota term is confusing as oVirt already has
> quota today, and cpu API has his own quota definition. So please try
> to come up with a different terminology.
>
> I like your idea of setting an initial value but I need some more time
> for it to come up with my insights.
> Also, I completely agree with your concept of letting mom handle
> it in host level. We need to verify it does not break anything related
> to SPM. This is something we need to synchronize with the storage guys.
>
> Looking into the engine area, we should start thinking on how this will
> be supported in the main storage entities and VM / template / instance
> entities. So you may want to add a section on this as well. This leads
> me to think of importing and exporting a VM which may want to maintain
> the defined IO QoS. Any thoughts around it?
>
> Doron
>
Hi Doron,
Thanks for your questions and insightful thoughts. They are really 
inspiring.

I updated the design in 
http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth .
This time, I add  storage I/O bandwidth control according to the quota 
design and the SLA tries to ensure the reserved bandwidth for vDisks.
It requires the engine  to make the tuning decision, since vm uses the 
SD volumes may  reside on different hosts and only engine can obtain the 
global information.
I think this design will not lead to problem when importing or exporting 
a VM.

I will appreciate if you can take a look at the new design.

Best regards,
Mei Liu (Rose)



From dfediuck at redhat.com  Mon Jun  3 10:47:21 2013
From: dfediuck at redhat.com (Doron Fediuck)
Date: Mon, 3 Jun 2013 06:47:21 -0400 (EDT)
Subject: oVirt @China
In-Reply-To: <999527024.17643542.1370255730449.JavaMail.root@redhat.com>
Message-ID: <649567228.17650319.1370256441003.JavaMail.root@redhat.com>

Hi everyone,
On our last visit to Shanghai we discovered there's no
access to YouTube, so oVirt users and developers in China
cannot see the cool video clips we have for oVirt.

To resolve that, we were able to open an account in YouKu
for oVirt, which will host the relevant video clips. I'm
happy to announce oVirt's YouKu page: 

http://i.youku.com/theovirt

Feel free to share with your friends!

Here's how you can help:

- People who do not see their video clips there can
  mail me and I'll upload it to YouKu.

- For new contents please contact me as well so I'll
  upload it to YouKu.

Special thanks to Jimmy and the Intel team, who
helped me with this effort. 

Looking forward to see more video clips,
Doron


From mpastern at redhat.com  Mon Jun  3 14:23:41 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Mon, 03 Jun 2013 17:23:41 +0300
Subject: Fwd: Re: edit storage connections
In-Reply-To: <51AC85B6.6070303@redhat.com>
References: <51AC85B6.6070303@redhat.com>
Message-ID: <51ACA6ED.3070907@redhat.com>


Hi Alissa,

On 06/02/2013 02:15 PM, Alissa Bonas wrote:
> Hi,
> 
> I started the page for the feature I mentioned to you last week.
> It's still not complete so I didn't distribute it yet, but I hope you can understand the concept - there's a REST part in the wiki too.
> If you have any comments/suggestions/improvements especially (but not only) on REST, I'd love to hear it.
> Thanks
> 
> http://www.ovirt.org/Features/Edit_Connection_Properties
> 
>
>
> REST
>
> In order to allow editing connections, a new root resource will be introduced
> that will allow add/edit/delete/get of connections to storage.
>
> TODO: model the new connection entity. Should represent several storage types.
>
>
> New connection (POST)
>
> It will be possible to create a new connection without adding a storage domain
> along with it, and later on create a storage domain and relate it to existing connection
> by providing the connection id.

please describe 'connection/s' resource/collection URLs/XML modelling
(iiuc XML modelling basically should remain the same)

>
> Delete connection
>
> Deletion of connection will be possible only if no storage domain is connected to it (orphan connection).
>
>
> Update existing connection (PUT)
>
> It will be possible to update connection details when storage domain connected to it is in
> maintenance state. Most of connection fields can be updated (such as path/iqn/address/port/user/password)
> - each storage type has its relevant set of fields, id of connection will be immutable.
>
> In addition, for each storage domain it should be possible to view (GET) its storage connections
> by approaching it via a specific subresource: /api/storagedomains/<storageDomainId>/connections

question: what about today's api? we can't break it, i'd suggest deprecating it and continue supporting
till we drop it, - it means having both:

1. connection details in the SD (active connection)
2. connections under /api/storagedomains/<storageDomainId>/connections + /api/connections



-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D




From liumbj at linux.vnet.ibm.com  Wed Jun  5 08:11:34 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Wed, 05 Jun 2013 16:11:34 +0800
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <51AC6FDE.5030007@linux.vnet.ibm.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
Message-ID: <51AEF2B6.6070503@linux.vnet.ibm.com>

Hi Doron,
After the discussion, we found that the last version of design is a 
little complex, so I simplified it and post it to

http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth

We want to let MOM in each host decides how to tune the bandwidth limit 
of vms in that host instead of letting engine to make the whole decision 
based on statistics from vms in diverse hosts. Maybe we can consider 
starting from the basic one in wiki.

Thanks & best regards,
Mei Liu(Rose)

-------- Original Message --------
Subject: 	Re: SLA feature for storage I/O bandwidth
Date: 	Mon, 03 Jun 2013 18:28:46 +0800
From: 	Mei Liu <liumbj at linux.vnet.ibm.com>
To: 	Doron Fediuck <dfediuck at redhat.com>
CC: 	arch at ovirt.org



On 05/29/2013 11:34 PM, Doron Fediuck wrote:
> ----- Original Message -----
>> From: "Mei Liu" <liumbj at linux.vnet.ibm.com>
>> To: "Dave Neary" <dneary at redhat.com>
>> Cc: arch at ovirt.org
>> Sent: Wednesday, May 29, 2013 11:35:12 AM
>> Subject: Re: SLA feature for storage I/O bandwidth
>>
>> On 05/29/2013 03:42 PM, Dave Neary wrote:
>>> Hi Mei Liu,
>>>
>>> On 05/28/2013 10:18 AM, Mei Liu wrote:
>>>> I created a drafted wiki page on  design of storage I/O bandwidth SLA in
>>>> the following link:
>>>>
>>>> http://www.ovirt.org/SLA_for_storage_resource .
>>>>
>>>> I will appreciate the efforts if anyone who works on ovirt engine, vdsm
>>>> or mom could give some comments. TIA.
>>> Just out of interest - which version of oVirt are you targeting this
>>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
>>> feature freeze (but we have a release meeting later to discuss that).
>>>
>>> Thanks,
>>> Dave.
>>>
>> Hi Dave,
>> The basic i/o tune functionality for vdsm is almost ready. However,
>> there is nothing written on the Engine side and no policy for automatic
>> tuning is applied yet.
>> I am not sure if the basic functionality can target 3.3.
>>
>>
>> Best regards,
>> Mei Liu
>>
> Hi Mey Liu,
> I'm still going over the wiki, but a few things we need to consider;
> First of all QoS for storage I/O bandwidth is a part of a larger SLA
> policy which may include network QoS, CPU and memory QoS and the quota
> we implement today.
>
> So first of all we need to make sure your design does not conflict
> the other QoS parts, which is what I'm looking into now.
>
> Additionally, using the quota term is confusing as oVirt already has
> quota today, and cpu API has his own quota definition. So please try
> to come up with a different terminology.
>
> I like your idea of setting an initial value but I need some more time
> for it to come up with my insights.
> Also, I completely agree with your concept of letting mom handle
> it in host level. We need to verify it does not break anything related
> to SPM. This is something we need to synchronize with the storage guys.
>
> Looking into the engine area, we should start thinking on how this will
> be supported in the main storage entities and VM / template / instance
> entities. So you may want to add a section on this as well. This leads
> me to think of importing and exporting a VM which may want to maintain
> the defined IO QoS. Any thoughts around it?
>
> Doron
>
Hi Doron,
Thanks for your questions and insightful thoughts. They are really
inspiring.

I updated the design in
http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth .
This time, I add  storage I/O bandwidth control according to the quota
design and the SLA tries to ensure the reserved bandwidth for vDisks.
It requires the engine  to make the tuning decision, since vm uses the
SD volumes may  reside on different hosts and only engine can obtain the
global information.
I think this design will not lead to problem when importing or exporting
a VM.

I will appreciate if you can take a look at the new design.

Best regards,
Mei Liu (Rose)

_______________________________________________
Arch mailing list
Arch at ovirt.org
http://lists.ovirt.org/mailman/listinfo/arch




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/f8b21d71/attachment.html>

From alonbl at redhat.com  Wed Jun  5 08:54:57 2013
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Wed, 5 Jun 2013 04:54:57 -0400 (EDT)
Subject: gerrit.ovirt.org is malfunctioning
In-Reply-To: <1102879578.6859021.1370422476663.JavaMail.root@redhat.com>
Message-ID: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>

Hi,

Last two/three days more and more internals errors are gotten out of gerrit during pushes.

Can someone have a look?

Thanks,
Alon


From eedri at redhat.com  Wed Jun  5 09:02:38 2013
From: eedri at redhat.com (Eyal Edri)
Date: Wed, 5 Jun 2013 05:02:38 -0400 (EDT)
Subject: gerrit.ovirt.org is malfunctioning
In-Reply-To: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>
References: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>
Message-ID: <1587105797.4071621.1370422958535.JavaMail.root@redhat.com>

I believe it might be related to the extra load being done in it from jenkins gerrit per patch jobs.

there a plan to upgrade it soon that might resolve it.

how often do you experience errors and what sort of errors?

Eyal.

----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "arch" <arch at ovirt.org>
> Sent: Wednesday, June 5, 2013 11:54:57 AM
> Subject: gerrit.ovirt.org is malfunctioning
> 
> Hi,
> 
> Last two/three days more and more internals errors are gotten out of gerrit
> during pushes.
> 
> Can someone have a look?
> 
> Thanks,
> Alon
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 


From amureini at redhat.com  Wed Jun  5 09:03:41 2013
From: amureini at redhat.com (Allon Mureinik)
Date: Wed, 5 Jun 2013 05:03:41 -0400 (EDT)
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <51AEF2B6.6070503@linux.vnet.ibm.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
Message-ID: <26148712.4071809.1370423021002.JavaMail.root@redhat.com>

Hi Mei, 

How are you treating shared disks? 
Is the limitation defined per disk (as a total), or per disk-vm relation? 

----- Original Message -----

> From: "Mei Liu" <liumbj at linux.vnet.ibm.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: arch at ovirt.org
> Sent: Wednesday, June 5, 2013 11:11:34 AM
> Subject: Re: SLA feature for storage I/O bandwidth

> Hi Doron,
> After the discussion, we found that the last version of design is a little
> complex, so I simplified it and post it to
> http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth
> We want to let MOM in each host decides how to tune the bandwidth limit of
> vms in that host instead of letting engine to make the whole decision based
> on statistics from vms in diverse hosts. Maybe we can consider starting from
> the basic one in wiki.

> Thanks & best regards,
> Mei Liu(Rose)

> -------- Original Message --------
> Subject: 	Re: SLA feature for storage I/O bandwidth
> Date: 	Mon, 03 Jun 2013 18:28:46 +0800
> From: 	Mei Liu <liumbj at linux.vnet.ibm.com>
> To: 	Doron Fediuck <dfediuck at redhat.com>
> CC: 	arch at ovirt.org

> On 05/29/2013 11:34 PM, Doron Fediuck wrote:
> > ----- Original Message -----
> >> From: "Mei Liu" <liumbj at linux.vnet.ibm.com> >> To: "Dave Neary"
> >> <dneary at redhat.com> >> Cc: arch at ovirt.org >> Sent: Wednesday, May 29,
> >> 2013 11:35:12 AM
> >> Subject: Re: SLA feature for storage I/O bandwidth
> >>
> >> On 05/29/2013 03:42 PM, Dave Neary wrote:
> >>> Hi Mei Liu,
> >>>
> >>> On 05/28/2013 10:18 AM, Mei Liu wrote:
> >>>> I created a drafted wiki page on  design of storage I/O bandwidth SLA in
> >>>> the following link:
> >>>>
> >>>> http://www.ovirt.org/SLA_for_storage_resource .
> >>>>
> >>>> I will appreciate the efforts if anyone who works on ovirt engine, vdsm
> >>>> or mom could give some comments. TIA.
> >>> Just out of interest - which version of oVirt are you targeting this
> >>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
> >>> feature freeze (but we have a release meeting later to discuss that).
> >>>
> >>> Thanks,
> >>> Dave.
> >>>
> >> Hi Dave,
> >> The basic i/o tune functionality for vdsm is almost ready. However,
> >> there is nothing written on the Engine side and no policy for automatic
> >> tuning is applied yet.
> >> I am not sure if the basic functionality can target 3.3.
> >>
> >>
> >> Best regards,
> >> Mei Liu
> >>
> > Hi Mey Liu,
> > I'm still going over the wiki, but a few things we need to consider;
> > First of all QoS for storage I/O bandwidth is a part of a larger SLA
> > policy which may include network QoS, CPU and memory QoS and the quota
> > we implement today.
> >
> > So first of all we need to make sure your design does not conflict
> > the other QoS parts, which is what I'm looking into now.
> >
> > Additionally, using the quota term is confusing as oVirt already has
> > quota today, and cpu API has his own quota definition. So please try
> > to come up with a different terminology.
> >
> > I like your idea of setting an initial value but I need some more time
> > for it to come up with my insights.
> > Also, I completely agree with your concept of letting mom handle
> > it in host level. We need to verify it does not break anything related
> > to SPM. This is something we need to synchronize with the storage guys.
> >
> > Looking into the engine area, we should start thinking on how this will
> > be supported in the main storage entities and VM / template / instance
> > entities. So you may want to add a section on this as well. This leads
> > me to think of importing and exporting a VM which may want to maintain
> > the defined IO QoS. Any thoughts around it?
> >
> > Doron
> >
> Hi Doron,
> Thanks for your questions and insightful thoughts. They are really
> inspiring.

> I updated the design in
> http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth .
> This time, I add  storage I/O bandwidth control according to the quota
> design and the SLA tries to ensure the reserved bandwidth for vDisks.
> It requires the engine  to make the tuning decision, since vm uses the
> SD volumes may  reside on different hosts and only engine can obtain the
> global information.
> I think this design will not lead to problem when importing or exporting
> a VM.

> I will appreciate if you can take a look at the new design.

> Best regards,
> Mei Liu (Rose)

> _______________________________________________
> Arch mailing list Arch at ovirt.org http://lists.ovirt.org/mailman/listinfo/arch

> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/d284d2e7/attachment.html>

From asegurap at redhat.com  Wed Jun  5 09:06:49 2013
From: asegurap at redhat.com (Antoni Segura Puimedon)
Date: Wed, 5 Jun 2013 05:06:49 -0400 (EDT)
Subject: gerrit.ovirt.org is malfunctioning
In-Reply-To: <1587105797.4071621.1370422958535.JavaMail.root@redhat.com>
References: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>
	<1587105797.4071621.1370422958535.JavaMail.root@redhat.com>
Message-ID: <1907450832.6861227.1370423209593.JavaMail.root@redhat.com>

This kind of error: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/1751/console
In more than 50% of the new patch submissions in vdsm.

----- Original Message -----
> From: "Eyal Edri" <eedri at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "arch" <arch at ovirt.org>
> Sent: Wednesday, June 5, 2013 11:02:38 AM
> Subject: Re: gerrit.ovirt.org is malfunctioning
> 
> I believe it might be related to the extra load being done in it from jenkins
> gerrit per patch jobs.
> 
> there a plan to upgrade it soon that might resolve it.
> 
> how often do you experience errors and what sort of errors?
> 
> Eyal.
> 
> ----- Original Message -----
> > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > To: "arch" <arch at ovirt.org>
> > Sent: Wednesday, June 5, 2013 11:54:57 AM
> > Subject: gerrit.ovirt.org is malfunctioning
> > 
> > Hi,
> > 
> > Last two/three days more and more internals errors are gotten out of gerrit
> > during pushes.
> > 
> > Can someone have a look?
> > 
> > Thanks,
> > Alon
> > _______________________________________________
> > Arch mailing list
> > Arch at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/arch
> > 
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 


From tnisan at redhat.com  Wed Jun  5 09:47:00 2013
From: tnisan at redhat.com (Tal Nisan)
Date: Wed, 05 Jun 2013 12:47:00 +0300
Subject: gerrit.ovirt.org is malfunctioning
In-Reply-To: <1907450832.6861227.1370423209593.JavaMail.root@redhat.com>
References: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>
	<1587105797.4071621.1370422958535.JavaMail.root@redhat.com>
	<1907450832.6861227.1370423209593.JavaMail.root@redhat.com>
Message-ID: <51AF0914.7070202@redhat.com>

Now it's getting stuck also when trying to click on a change link:

Application Error
Server Error
org.eclipse.jgit.errors.MissingObjectException: Missing unknown 
d77d2118ec831152f7f4c3d019d6367a2a615e9b


On 06/05/2013 12:06 PM, Antoni Segura Puimedon wrote:
> This kind of error: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/1751/console
> In more than 50% of the new patch submissions in vdsm.
>
> ----- Original Message -----
>> From: "Eyal Edri" <eedri at redhat.com>
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: "arch" <arch at ovirt.org>
>> Sent: Wednesday, June 5, 2013 11:02:38 AM
>> Subject: Re: gerrit.ovirt.org is malfunctioning
>>
>> I believe it might be related to the extra load being done in it from jenkins
>> gerrit per patch jobs.
>>
>> there a plan to upgrade it soon that might resolve it.
>>
>> how often do you experience errors and what sort of errors?
>>
>> Eyal.
>>
>> ----- Original Message -----
>>> From: "Alon Bar-Lev" <alonbl at redhat.com>
>>> To: "arch" <arch at ovirt.org>
>>> Sent: Wednesday, June 5, 2013 11:54:57 AM
>>> Subject: gerrit.ovirt.org is malfunctioning
>>>
>>> Hi,
>>>
>>> Last two/three days more and more internals errors are gotten out of gerrit
>>> during pushes.
>>>
>>> Can someone have a look?
>>>
>>> Thanks,
>>> Alon
>>> _______________________________________________
>>> Arch mailing list
>>> Arch at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/arch
>>>
>> _______________________________________________
>> Arch mailing list
>> Arch at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/arch
>>
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/2072180e/attachment.html>

From eedri at redhat.com  Wed Jun  5 09:54:42 2013
From: eedri at redhat.com (Eyal Edri)
Date: Wed, 5 Jun 2013 05:54:42 -0400 (EDT)
Subject: gerrit.ovirt.org is malfunctioning
In-Reply-To: <1907450832.6861227.1370423209593.JavaMail.root@redhat.com>
References: <2002414152.6859035.1370422497381.JavaMail.root@redhat.com>
	<1587105797.4071621.1370422958535.JavaMail.root@redhat.com>
	<1907450832.6861227.1370423209593.JavaMail.root@redhat.com>
Message-ID: <747720947.4094202.1370426082030.JavaMail.root@redhat.com>

[ cc'ing infra ]

i disabled the findbugs per-patch job for now,
and i'll restart the gerrit service via the jenkins job.

let' hope it will do the trick.

Eyal.

----- Original Message -----
> From: "Antoni Segura Puimedon" <asegurap at redhat.com>
> To: "Eyal Edri" <eedri at redhat.com>
> Cc: "Alon Bar-Lev" <alonbl at redhat.com>, "arch" <arch at ovirt.org>
> Sent: Wednesday, June 5, 2013 12:06:49 PM
> Subject: Re: gerrit.ovirt.org is malfunctioning
> 
> This kind of error:
> http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/1751/console
> In more than 50% of the new patch submissions in vdsm.
> 
> ----- Original Message -----
> > From: "Eyal Edri" <eedri at redhat.com>
> > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > Cc: "arch" <arch at ovirt.org>
> > Sent: Wednesday, June 5, 2013 11:02:38 AM
> > Subject: Re: gerrit.ovirt.org is malfunctioning
> > 
> > I believe it might be related to the extra load being done in it from
> > jenkins
> > gerrit per patch jobs.
> > 
> > there a plan to upgrade it soon that might resolve it.
> > 
> > how often do you experience errors and what sort of errors?
> > 
> > Eyal.
> > 
> > ----- Original Message -----
> > > From: "Alon Bar-Lev" <alonbl at redhat.com>
> > > To: "arch" <arch at ovirt.org>
> > > Sent: Wednesday, June 5, 2013 11:54:57 AM
> > > Subject: gerrit.ovirt.org is malfunctioning
> > > 
> > > Hi,
> > > 
> > > Last two/three days more and more internals errors are gotten out of
> > > gerrit
> > > during pushes.
> > > 
> > > Can someone have a look?
> > > 
> > > Thanks,
> > > Alon
> > > _______________________________________________
> > > Arch mailing list
> > > Arch at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/arch
> > > 
> > _______________________________________________
> > Arch mailing list
> > Arch at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/arch
> > 
> 


From liumbj at linux.vnet.ibm.com  Wed Jun  5 10:14:31 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Wed, 05 Jun 2013 18:14:31 +0800
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <26148712.4071809.1370423021002.JavaMail.root@redhat.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
	<26148712.4071809.1370423021002.JavaMail.root@redhat.com>
Message-ID: <51AF0F87.2000809@linux.vnet.ibm.com>

On 06/05/2013 05:03 PM, Allon Mureinik wrote:
> Hi Mei,
>
> How are you treating shared disks?
> Is the limitation defined per disk (as a total), or per disk-vm relation?
>
The existing API in libvirt is to limit the  per vm's vDisk. e.g. hda of VM1

So the limitation is on per vm's vDisk, if we don't consider quota for 
SD in the design (the basic design). That limitation value is adjusted 
based on if backend storage(localfs, nfs, glusterfs...) is heavily used.

SD's IO bandwidth quota is to constrain the sum of vDisk limitation's 
lower bound(min value of limitation, it is similar to the concept of 
reserving). These vDisks use the volumes in the SD.


> ------------------------------------------------------------------------
>
>     *From: *"Mei Liu" <liumbj at linux.vnet.ibm.com>
>     *To: *"Doron Fediuck" <dfediuck at redhat.com>
>     *Cc: *arch at ovirt.org
>     *Sent: *Wednesday, June 5, 2013 11:11:34 AM
>     *Subject: *Re: SLA feature for storage I/O bandwidth
>
>     Hi Doron,
>     After the discussion, we found that the last version of design is
>     a little complex, so I simplified it and post it to
>
>     http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth
>
>     We want to let MOM in each host decides how to tune the bandwidth
>     limit of vms in that host instead of letting engine to make the
>     whole decision based on statistics from vms in diverse hosts.
>     Maybe we can consider starting from the basic one in wiki.
>
>     Thanks & best regards,
>     Mei Liu(Rose)
>
>     -------- Original Message --------
>     Subject: 	Re: SLA feature for storage I/O bandwidth
>     Date: 	Mon, 03 Jun 2013 18:28:46 +0800
>     From: 	Mei Liu <liumbj at linux.vnet.ibm.com>
>     To: 	Doron Fediuck <dfediuck at redhat.com>
>     CC: 	arch at ovirt.org
>
>
>
>     On 05/29/2013 11:34 PM, Doron Fediuck wrote:
>     > ----- Original Message -----
>     >> From: "Mei Liu"<liumbj at linux.vnet.ibm.com>
>     >> To: "Dave Neary"<dneary at redhat.com>
>     >> Cc:arch at ovirt.org
>     >> Sent: Wednesday, May 29, 2013 11:35:12 AM
>     >> Subject: Re: SLA feature for storage I/O bandwidth
>     >>
>     >> On 05/29/2013 03:42 PM, Dave Neary wrote:
>     >>> Hi Mei Liu,
>     >>>
>     >>> On 05/28/2013 10:18 AM, Mei Liu wrote:
>     >>>> I created a drafted wiki page on  design of storage I/O bandwidth SLA in
>     >>>> the following link:
>     >>>>
>     >>>>http://www.ovirt.org/SLA_for_storage_resource  .
>     >>>>
>     >>>> I will appreciate the efforts if anyone who works on ovirt engine, vdsm
>     >>>> or mom could give some comments. TIA.
>     >>> Just out of interest - which version of oVirt are you targeting this
>     >>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
>     >>> feature freeze (but we have a release meeting later to discuss that).
>     >>>
>     >>> Thanks,
>     >>> Dave.
>     >>>
>     >> Hi Dave,
>     >> The basic i/o tune functionality for vdsm is almost ready. However,
>     >> there is nothing written on the Engine side and no policy for automatic
>     >> tuning is applied yet.
>     >> I am not sure if the basic functionality can target 3.3.
>     >>
>     >>
>     >> Best regards,
>     >> Mei Liu
>     >>
>     > Hi Mey Liu,
>     > I'm still going over the wiki, but a few things we need to consider;
>     > First of all QoS for storage I/O bandwidth is a part of a larger SLA
>     > policy which may include network QoS, CPU and memory QoS and the quota
>     > we implement today.
>     >
>     > So first of all we need to make sure your design does not conflict
>     > the other QoS parts, which is what I'm looking into now.
>     >
>     > Additionally, using the quota term is confusing as oVirt already has
>     > quota today, and cpu API has his own quota definition. So please try
>     > to come up with a different terminology.
>     >
>     > I like your idea of setting an initial value but I need some more time
>     > for it to come up with my insights.
>     > Also, I completely agree with your concept of letting mom handle
>     > it in host level. We need to verify it does not break anything related
>     > to SPM. This is something we need to synchronize with the storage guys.
>     >
>     > Looking into the engine area, we should start thinking on how this will
>     > be supported in the main storage entities and VM / template / instance
>     > entities. So you may want to add a section on this as well. This leads
>     > me to think of importing and exporting a VM which may want to maintain
>     > the defined IO QoS. Any thoughts around it?
>     >
>     > Doron
>     >
>     Hi Doron,
>     Thanks for your questions and insightful thoughts. They are really
>     inspiring.
>
>     I updated the design in
>     http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth  .
>     This time, I add  storage I/O bandwidth control according to the quota
>     design and the SLA tries to ensure the reserved bandwidth for vDisks.
>     It requires the engine  to make the tuning decision, since vm uses the
>     SD volumes may  reside on different hosts and only engine can obtain the
>     global information.
>     I think this design will not lead to problem when importing or exporting
>     a VM.
>
>     I will appreciate if you can take a look at the new design.
>
>     Best regards,
>     Mei Liu (Rose)
>
>     _______________________________________________
>     Arch mailing list
>     Arch at ovirt.org
>     http://lists.ovirt.org/mailman/listinfo/arch
>
>
>
>
>
>     _______________________________________________
>     Arch mailing list
>     Arch at ovirt.org
>     http://lists.ovirt.org/mailman/listinfo/arch
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/04441b3a/attachment.html>

From mpastern at redhat.com  Wed Jun  5 10:28:02 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 05 Jun 2013 13:28:02 +0300
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51A76B36.7030408@linux.vnet.ibm.com>
References: <51A76B36.7030408@linux.vnet.ibm.com>
Message-ID: <51AF12B2.3000903@redhat.com>


Hi Deepak,

On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
> Hi All,
>     Per the previous mail from Ayal.. I am refering to the flow below...
> 
> The backup flow should be (all api calls are engine REST API):

I disagree with this, by following this pattern you move internal
logic (even more than this given implementation specific logic) to
the api users instead of abstracting it and creating solid api for
backups,

+++++++++ several api calls for single backup ++++++++++++++


cons:
====

1. user have to be aware of internal (implementation specific) flow of each provider in order to be able creating backup
   1.1 call add(a)->do(a)->del(a)->do(b)->etc.
2. no flexibility for backup provider to change the backup flow (as it would be considered an api break)
3. no atomic resolution for the /backup failures
   3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
        that user have to del(a) manually and if it's deeper in the flow
        every api user will have to implement own rollbacks to the steps that took
        place before the failure.
4. backward compatibility becomes an issue when exposing internals in such a low level.
5. new features expose becomes much complex task cause they may require performing extra
   step/s in a middle.

pros:
====

can't think of any ..., cause it doesn't give you flexibility, but creates unnatural
(to public APIs) complexity.


+++++++++ single api call for single backup ++++++++++++++

cons:
====

nothing comes to my mind, are there any?

pros:
====

1. user is not aware of any backup internals and can consume different /backup
   providers using very same api and supplying parameters relevant for the provider
   he'd like to use.
2. backup providers can change/extend internal flow as much as they want/need, only
   by modifying internal engine driver, user using abstraction over the backup api
   won't feel the difference.
3. backup providers can execute backup as single atomic operation and take care
   for rollback cleanups in the transaction.
4. backup providers can easily maintain backward compatibility over abstraction
5. adding new features that require flow extension can be easily hidden under
   the abstraction layer.

also this is a common practice to expose abstraction layer to public, while
vendors only have to implement the driver in order to be supported.


> 1. create vm snapshot (the api call exists today)
> 2. get VM Config (new API)
> 3. prepare backup disk (new api, should probably accept: hostid, disk;
> return: paths to device on host as well as map of changed blocks) -
> this should be called for every disk that needs to be backed up. Note
> that VM snapshot takes a snap of *all* disks of the VM
> 4. attach disk to backup vm (the api call exists today.  This assumes
> virt app) - also has to be called per disk to back up
> 5. At this point backup app would do the backup
> 

this can be easily done under the single api call:
=================================================

POST /api/vms/xxx/disks/yyy/backups

<backup>
  <host id=aaa> #3
  <backup_vm id=bbb> #1, #2
  <delete_snapshot>true|false<delete_snapshot/> #7
<backup/>

1. create vm snapshot => you already in context of vm, trigger snapshot on it
2. get VM Config => you already in context of vm, collect relevant meta
3. prepare backup disk => you in context of the disk already, do whatever is needed
4. attach disk to backup vm => you have backup_vm id in the request body
5. do the actual backup

does this makes sense?

> 
> 5) detach disk (symmetric to 4)
> 6. teardown backup disk (symmetric to 3)
> 7. delete snap - This can only be called if VM is down today and is
> not mandatory in any event.  Once we have live merge, it should be
> policy driven (user should be able to choose whether to keep or delete)
> 
> Questions...
> 
> 1) My current implmn of prepareBackupDisk in VDSM (pls see http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
> returns drive and cbtinfo as dicts back to engine. The understnading here is that the drive returned will be a dict which has all the necessary info
> to represent a NBD drive to libvirt... hence this drive will be passed as-is in the "attach disk to VM" REST API. Is this correct ?
> 
> Note that i cannot return a disk UUID (like VDSM does for create volume case).. because in preparebackupdisk case the image is exported usign qemu-nbd
> as a block device over the network... and hence its not a regular vdsm-type disk, hence no UUID, Agree ?
> 
> 2) In "attach disk to VM" REST API user will pass the drive dict which he/she got as part of preparebackupdisk api.. as-is
> and VDSM will need to add support for accepting NBD disk as a valid disktype in VM.hotplugDisk() code of VDSM.
> 
> The ability to add a network disk is present in libvirtvm.py... as part of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
> but it doesn't work at the API.py level.. its only from prepareImage onwards... hence the need to modify API.py to accept NBD disk type and connect
> to the libvirtvm.py appropriately
> 
> Is this acceptable.. otherwise there is no good way to pass the NBD drive's info back to VDSM as part of existing "attach disk to VM" API
> Also does the existing "attach disk to VM" API work if a pre-loaded drive dict/hashmap is provided by the user. This will have drive->type = network
> and not file/blcok as is currently supported
> 
> 3) After "attach disk to backup VM" REST API step.. the understnading is that some backup vendor specific API will be called to tell the
> backup appln....
>     -- Block device (attached to the VM) to be used as src for backup
>     -- CBT info of this blcok device (which was recd. as part of prepareBackupDisk API)
> Is this correct ?
> 
> thanx,
> deepak
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From mpastern at redhat.com  Wed Jun  5 10:31:57 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 05 Jun 2013 13:31:57 +0300
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF12B2.3000903@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
Message-ID: <51AF139D.8060807@redhat.com>

On 06/05/2013 01:28 PM, Michael Pasternak wrote:
> 
> Hi Deepak,
> 
> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>> Hi All,
>>     Per the previous mail from Ayal.. I am refering to the flow below...
>>
>> The backup flow should be (all api calls are engine REST API):
> 
> I disagree with this, by following this pattern you move internal
> logic (even more than this given implementation specific logic) to
> the api users instead of abstracting it and creating solid api for
> backups,
> 
> +++++++++ several api calls for single backup ++++++++++++++
> 
> 
> cons:
> ====
> 
> 1. user have to be aware of internal (implementation specific) flow of each provider in order to be able creating backup
>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
> 2. no flexibility for backup provider to change the backup flow (as it would be considered an api break)
> 3. no atomic resolution for the /backup failures
>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>         that user have to del(a) manually and if it's deeper in the flow
>         every api user will have to implement own rollbacks to the steps that took
>         place before the failure.
> 4. backward compatibility becomes an issue when exposing internals in such a low level.
> 5. new features expose becomes much complex task cause they may require performing extra
>    step/s in a middle.
> 
> pros:
> ====
> 
> can't think of any ..., cause it doesn't give you flexibility, but creates unnatural
> (to public APIs) complexity.
> 
> 
> +++++++++ single api call for single backup ++++++++++++++
> 
> cons:
> ====
> 
> nothing comes to my mind, are there any?
> 
> pros:
> ====
> 
> 1. user is not aware of any backup internals and can consume different /backup
>    providers using very same api and supplying parameters relevant for the provider
>    he'd like to use.
> 2. backup providers can change/extend internal flow as much as they want/need, only
>    by modifying internal engine driver, user using abstraction over the backup api
>    won't feel the difference.
> 3. backup providers can execute backup as single atomic operation and take care
>    for rollback cleanups in the transaction.
> 4. backup providers can easily maintain backward compatibility over abstraction
> 5. adding new features that require flow extension can be easily hidden under
>    the abstraction layer.
> 
> also this is a common practice to expose abstraction layer to public, while
> vendors only have to implement the driver in order to be supported.
> 
> 
>> 1. create vm snapshot (the api call exists today)
>> 2. get VM Config (new API)
>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
>> return: paths to device on host as well as map of changed blocks) -
>> this should be called for every disk that needs to be backed up. Note
>> that VM snapshot takes a snap of *all* disks of the VM
>> 4. attach disk to backup vm (the api call exists today.  This assumes
>> virt app) - also has to be called per disk to back up
>> 5. At this point backup app would do the backup
>>
> 
> this can be easily done under the single api call:
> =================================================
> 
> POST /api/vms/xxx/disks/yyy/backups
> 
> <backup>
>   <host id=aaa> #3
>   <backup_vm id=bbb> #1, #2

typo ^, should be: <backup_vm id=bbb> #4

>   <delete_snapshot>true|false<delete_snapshot/> #7
> <backup/>
> 
> 1. create vm snapshot => you already in context of vm, trigger snapshot on it
> 2. get VM Config => you already in context of vm, collect relevant meta
> 3. prepare backup disk => you in context of the disk already, do whatever is needed
> 4. attach disk to backup vm => you have backup_vm id in the request body
> 5. do the actual backup
> 
> does this makes sense?
> 
>>
>> 5) detach disk (symmetric to 4)
>> 6. teardown backup disk (symmetric to 3)
>> 7. delete snap - This can only be called if VM is down today and is
>> not mandatory in any event.  Once we have live merge, it should be
>> policy driven (user should be able to choose whether to keep or delete)
>>
>> Questions...
>>
>> 1) My current implmn of prepareBackupDisk in VDSM (pls see http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>> returns drive and cbtinfo as dicts back to engine. The understnading here is that the drive returned will be a dict which has all the necessary info
>> to represent a NBD drive to libvirt... hence this drive will be passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>
>> Note that i cannot return a disk UUID (like VDSM does for create volume case).. because in preparebackupdisk case the image is exported usign qemu-nbd
>> as a block device over the network... and hence its not a regular vdsm-type disk, hence no UUID, Agree ?
>>
>> 2) In "attach disk to VM" REST API user will pass the drive dict which he/she got as part of preparebackupdisk api.. as-is
>> and VDSM will need to add support for accepting NBD disk as a valid disktype in VM.hotplugDisk() code of VDSM.
>>
>> The ability to add a network disk is present in libvirtvm.py... as part of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
>> but it doesn't work at the API.py level.. its only from prepareImage onwards... hence the need to modify API.py to accept NBD disk type and connect
>> to the libvirtvm.py appropriately
>>
>> Is this acceptable.. otherwise there is no good way to pass the NBD drive's info back to VDSM as part of existing "attach disk to VM" API
>> Also does the existing "attach disk to VM" API work if a pre-loaded drive dict/hashmap is provided by the user. This will have drive->type = network
>> and not file/blcok as is currently supported
>>
>> 3) After "attach disk to backup VM" REST API step.. the understnading is that some backup vendor specific API will be called to tell the
>> backup appln....
>>     -- Block device (attached to the VM) to be used as src for backup
>>     -- CBT info of this blcok device (which was recd. as part of prepareBackupDisk API)
>> Is this correct ?
>>
>> thanx,
>> deepak
>>
> 
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From deepakcs at linux.vnet.ibm.com  Wed Jun  5 11:14:42 2013
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Wed, 05 Jun 2013 16:44:42 +0530
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF12B2.3000903@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
Message-ID: <51AF1DA2.5070305@linux.vnet.ibm.com>

On 06/05/2013 03:58 PM, Michael Pasternak wrote:
> Hi Deepak,
>
> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>> Hi All,
>>      Per the previous mail from Ayal.. I am refering to the flow below...
>>
>> The backup flow should be (all api calls are engine REST API):
> I disagree with this, by following this pattern you move internal
> logic (even more than this given implementation specific logic) to
> the api users instead of abstracting it and creating solid api for
> backups,
>
> +++++++++ several api calls for single backup ++++++++++++++
>
>
> cons:
> ====
>
> 1. user have to be aware of internal (implementation specific) flow of each provider in order to be able creating backup
>     1.1 call add(a)->do(a)->del(a)->do(b)->etc.
> 2. no flexibility for backup provider to change the backup flow (as it would be considered an api break)
> 3. no atomic resolution for the /backup failures
>     3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>          that user have to del(a) manually and if it's deeper in the flow
>          every api user will have to implement own rollbacks to the steps that took
>          place before the failure.
> 4. backward compatibility becomes an issue when exposing internals in such a low level.
> 5. new features expose becomes much complex task cause they may require performing extra
>     step/s in a middle.
>
> pros:
> ====
>
> can't think of any ..., cause it doesn't give you flexibility, but creates unnatural
> (to public APIs) complexity.

I was able to dig the below from one of the older mails. I had exactly 
this Q on why we need separate APIs Vs 1 and this is what Ayal had to say

Deepak asked:

/Wouldn't it be better if there was just 1 REST API for startign backup
which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
internally caused the above individual APIs to get called ?
Why do we have to expose each of the steps in backup process as
individual REST APIs ?
/

Ayal said:

/1. Because a single API assumes that I always want to backup everything which isn't necessarily true (normally backup policy for system disk and data disks are different)
2. Going forward we will have disk level snapshot in which case the backup API would have to change.
3. You could always return the vm config per disk when calling "prepare backup disk" which would be a bit redundant now but once we have disk level snapshots it would be more relevant and this way remove 1 API call now.
Later on when we do have disk level snaps it could be an option to the command to take the snap or something I guess.
/
So i kind of agreed to what Ayal said... from the perspective that it provides flexibility to the end user using the API, as to how he/she wants to script/use it.
What do you think ?  I just wanted to put the above here... so that we are sure we can considering all aspects before making the decision on 1 Vs many APIs.

1 API provides simplicity, but getting flexibility using 1 API would mean passing addnl params.. for eg> If i want to backup a particular vmdisk only, then the xml below would change, rite ?
But looks like either way, its possible to get things done...
I personally don't have much experience in the area of REST APIs to take the call on whether we need 1 or multiple APIs... so I am still inconclusive here (sorry about that!)
All i can say is that with 1 API approach too, we can get flexibility.


>
>
> +++++++++ single api call for single backup ++++++++++++++
>
> cons:
> ====
>
> nothing comes to my mind, are there any?
>
> pros:
> ====
>
> 1. user is not aware of any backup internals and can consume different /backup
>     providers using very same api and supplying parameters relevant for the provider
>     he'd like to use.
> 2. backup providers can change/extend internal flow as much as they want/need, only
>     by modifying internal engine driver, user using abstraction over the backup api
>     won't feel the difference.
> 3. backup providers can execute backup as single atomic operation and take care
>     for rollback cleanups in the transaction.
> 4. backup providers can easily maintain backward compatibility over abstraction
> 5. adding new features that require flow extension can be easily hidden under
>     the abstraction layer.
>
> also this is a common practice to expose abstraction layer to public, while
> vendors only have to implement the driver in order to be supported.
>
>
>> 1. create vm snapshot (the api call exists today)
>> 2. get VM Config (new API)
>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
>> return: paths to device on host as well as map of changed blocks) -
>> this should be called for every disk that needs to be backed up. Note
>> that VM snapshot takes a snap of *all* disks of the VM
>> 4. attach disk to backup vm (the api call exists today.  This assumes
>> virt app) - also has to be called per disk to back up
>> 5. At this point backup app would do the backup
>>
> this can be easily done under the single api call:
> =================================================
>
> POST /api/vms/xxx/disks/yyy/backups
>
> <backup>
>    <host id=aaa> #3
>    <backup_vm id=bbb> #1, #2
>    <delete_snapshot>true|false<delete_snapshot/> #7
> <backup/>
>
> 1. create vm snapshot => you already in context of vm, trigger snapshot on it
> 2. get VM Config => you already in context of vm, collect relevant meta
> 3. prepare backup disk => you in context of the disk already, do whatever is needed
> 4. attach disk to backup vm => you have backup_vm id in the request body
> 5. do the actual backup
>
> does this makes sense?
>
>> 5) detach disk (symmetric to 4)
>> 6. teardown backup disk (symmetric to 3)
>> 7. delete snap - This can only be called if VM is down today and is
>> not mandatory in any event.  Once we have live merge, it should be
>> policy driven (user should be able to choose whether to keep or delete)
>>
>> Questions...
>>
>> 1) My current implmn of prepareBackupDisk in VDSM (pls see http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>> returns drive and cbtinfo as dicts back to engine. The understnading here is that the drive returned will be a dict which has all the necessary info
>> to represent a NBD drive to libvirt... hence this drive will be passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>
>> Note that i cannot return a disk UUID (like VDSM does for create volume case).. because in preparebackupdisk case the image is exported usign qemu-nbd
>> as a block device over the network... and hence its not a regular vdsm-type disk, hence no UUID, Agree ?
>>
>> 2) In "attach disk to VM" REST API user will pass the drive dict which he/she got as part of preparebackupdisk api.. as-is
>> and VDSM will need to add support for accepting NBD disk as a valid disktype in VM.hotplugDisk() code of VDSM.

>>
>> The ability to add a network disk is present in libvirtvm.py... as part of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
>> but it doesn't work at the API.py level.. its only from prepareImage onwards... hence the need to modify API.py to accept NBD disk type and connect
>> to the libvirtvm.py appropriately
>>
>> Is this acceptable.. otherwise there is no good way to pass the NBD drive's info back to VDSM as part of existing "attach disk to VM" API
>> Also does the existing "attach disk to VM" API work if a pre-loaded drive dict/hashmap is provided by the user. This will have drive->type = network
>> and not file/blcok as is currently supported

Mike,
     Could you provide your views on this pls ? How do we represent the 
NBD disk and pass it back to VDSM as part of 'hotplugDisk' API
Currently hotplugDisk doesn't support drive of type = network!

>>
>> 3) After "attach disk to backup VM" REST API step.. the understnading is that some backup vendor specific API will be called to tell the
>> backup appln....
>>      -- Block device (attached to the VM) to be used as src for backup
>>      -- CBT info of this blcok device (which was recd. as part of prepareBackupDisk API)
>> Is this correct ?
>>
>> thanx,
>> deepak
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/9d629164/attachment.html>

From mpastern at redhat.com  Wed Jun  5 11:48:13 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Wed, 05 Jun 2013 14:48:13 +0300
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF1DA2.5070305@linux.vnet.ibm.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com>
Message-ID: <51AF257D.6040800@redhat.com>

On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
>> Hi Deepak,
>>
>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>>> Hi All,
>>>     Per the previous mail from Ayal.. I am refering to the flow below...
>>>
>>> The backup flow should be (all api calls are engine REST API):
>> I disagree with this, by following this pattern you move internal
>> logic (even more than this given implementation specific logic) to
>> the api users instead of abstracting it and creating solid api for
>> backups,
>>
>> +++++++++ several api calls for single backup ++++++++++++++
>>
>>
>> cons:
>> ====
>>
>> 1. user have to be aware of internal (implementation specific) flow of each provider in order to be able creating backup
>>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
>> 2. no flexibility for backup provider to change the backup flow (as it would be considered an api break)
>> 3. no atomic resolution for the /backup failures
>>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>>         that user have to del(a) manually and if it's deeper in the flow
>>         every api user will have to implement own rollbacks to the steps that took
>>         place before the failure.
>> 4. backward compatibility becomes an issue when exposing internals in such a low level.
>> 5. new features expose becomes much complex task cause they may require performing extra
>>    step/s in a middle.
>>
>> pros:
>> ====
>>
>> can't think of any ..., cause it doesn't give you flexibility, but creates unnatural
>> (to public APIs) complexity.
> 
> I was able to dig the below from one of the older mails. I had exactly this Q on why we need separate APIs Vs 1 and this is what Ayal had to say
> 
> Deepak asked:
> 
> /Wouldn't it be better if there was just 1 REST API for startign backup
> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
> internally caused the above individual APIs to get called ?
> Why do we have to expose each of the steps in backup process as
> individual REST APIs ?
> /
> 
> Ayal said:
> 
> /1. Because a single API assumes that I always want to backup everything which isn't necessarily true (normally backup policy for system disk and data disks are different)

modelling backup under /api/vms/xxx/disks/yyy/backups solves this.

> 2. Going forward we will have disk level snapshot in which case the backup API would have to change.

same

> 3. You could always return the vm config per disk when calling "prepare backup disk" which would be a bit redundant now but once we have disk level snapshots it would be more relevant and this way remove 1 API call now.
> Later on when we do have disk level snaps it could be an option to the command to take the snap or something I guess.

it's only proves my point, we can/should hide this under api abstraction
rather than forcing users using different flows now and then.

> /
> So i kind of agreed to what Ayal said... from the perspective that it provides flexibility to the end user using the API, as to how he/she wants to script/use it.
> What do you think ?  I just wanted to put the above here... so that we are sure we can considering all aspects before making the decision on 1 Vs many APIs.

i've mentioned this before, - i don't think this is about flexibility, if you want to allow
user doing several things differently, - expose several api for same thing (like method overloads),
but it doesn't mean that we should be exposing implementation internals to end user.

> 
> 1 API provides simplicity, but getting flexibility using 1 API would mean passing addnl params.. for eg> If i want to backup a particular vmdisk only, then the xml below would change, rite ?

no, you misunderstood, see url [1] from my previous eamil, you already doing this for specific disk - yyy

[1] /api/vms/xxx/disks/yyy/backups

> But looks like either way, its possible to get things done... 
> I personally don't have much experience in the area of REST APIs to take the call on whether we need 1 or multiple APIs... so I am still inconclusive here (sorry about that!)
> All i can say is that with 1 API approach too, we can get flexibility.
> 
> 
>>
>>
>> +++++++++ single api call for single backup ++++++++++++++
>>
>> cons:
>> ====
>>
>> nothing comes to my mind, are there any?
>>
>> pros:
>> ====
>>
>> 1. user is not aware of any backup internals and can consume different /backup
>>    providers using very same api and supplying parameters relevant for the provider
>>    he'd like to use.
>> 2. backup providers can change/extend internal flow as much as they want/need, only
>>    by modifying internal engine driver, user using abstraction over the backup api
>>    won't feel the difference.
>> 3. backup providers can execute backup as single atomic operation and take care
>>    for rollback cleanups in the transaction.
>> 4. backup providers can easily maintain backward compatibility over abstraction
>> 5. adding new features that require flow extension can be easily hidden under
>>    the abstraction layer.
>>
>> also this is a common practice to expose abstraction layer to public, while
>> vendors only have to implement the driver in order to be supported.
>>
>>
>>> 1. create vm snapshot (the api call exists today)
>>> 2. get VM Config (new API)
>>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
>>> return: paths to device on host as well as map of changed blocks) -
>>> this should be called for every disk that needs to be backed up. Note
>>> that VM snapshot takes a snap of *all* disks of the VM
>>> 4. attach disk to backup vm (the api call exists today.  This assumes
>>> virt app) - also has to be called per disk to back up
>>> 5. At this point backup app would do the backup
>>>
>> this can be easily done under the single api call:
>> =================================================
>>
>> POST /api/vms/xxx/disks/yyy/backups
>>
>> <backup>
>>   <host id=aaa> #3
>>   <backup_vm id=bbb> #1, #2
>>   <delete_snapshot>true|false<delete_snapshot/> #7
>> <backup/>
>>
>> 1. create vm snapshot => you already in context of vm, trigger snapshot on it
>> 2. get VM Config => you already in context of vm, collect relevant meta
>> 3. prepare backup disk => you in context of the disk already, do whatever is needed
>> 4. attach disk to backup vm => you have backup_vm id in the request body
>> 5. do the actual backup
>>
>> does this makes sense?
>>
>>> 5) detach disk (symmetric to 4)
>>> 6. teardown backup disk (symmetric to 3)
>>> 7. delete snap - This can only be called if VM is down today and is
>>> not mandatory in any event.  Once we have live merge, it should be
>>> policy driven (user should be able to choose whether to keep or delete)
>>>
>>> Questions...
>>>
>>> 1) My current implmn of prepareBackupDisk in VDSM (pls see http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>>> returns drive and cbtinfo as dicts back to engine. The understnading here is that the drive returned will be a dict which has all the necessary info
>>> to represent a NBD drive to libvirt... hence this drive will be passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>>
>>> Note that i cannot return a disk UUID (like VDSM does for create volume case).. because in preparebackupdisk case the image is exported usign qemu-nbd
>>> as a block device over the network... and hence its not a regular vdsm-type disk, hence no UUID, Agree ?
>>>
>>> 2) In "attach disk to VM" REST API user will pass the drive dict which he/she got as part of preparebackupdisk api.. as-is
>>> and VDSM will need to add support for accepting NBD disk as a valid disktype in VM.hotplugDisk() code of VDSM.
> 
>>>
>>> The ability to add a network disk is present in libvirtvm.py... as part of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
>>> but it doesn't work at the API.py level.. its only from prepareImage onwards... hence the need to modify API.py to accept NBD disk type and connect
>>> to the libvirtvm.py appropriately
>>>
>>> Is this acceptable.. otherwise there is no good way to pass the NBD drive's info back to VDSM as part of existing "attach disk to VM" API
>>> Also does the existing "attach disk to VM" API work if a pre-loaded drive dict/hashmap is provided by the user. This will have drive->type = network
>>> and not file/blcok as is currently supported
> 
> Mike,
>     Could you provide your views on this pls ? How do we represent the NBD disk and pass it back to VDSM as part of 'hotplugDisk' API
> Currently hotplugDisk doesn't support drive of type = network!
> 
>>>
>>> 3) After "attach disk to backup VM" REST API step.. the understnading is that some backup vendor specific API will be called to tell the
>>> backup appln....
>>>     -- Block device (attached to the VM) to be used as src for backup
>>>     -- CBT info of this blcok device (which was recd. as part of prepareBackupDisk API)
>>> Is this correct ?
>>>
>>> thanx,
>>> deepak
>>>
>>
> 


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From deepakcs at linux.vnet.ibm.com  Wed Jun  5 12:00:56 2013
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Wed, 05 Jun 2013 17:30:56 +0530
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF257D.6040800@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
Message-ID: <51AF2878.1080500@linux.vnet.ibm.com>

On 06/05/2013 05:18 PM, Michael Pasternak wrote:
> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
>> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
>>> Hi Deepak,
>>>
>>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>>>> Hi All,
>>>>      Per the previous mail from Ayal.. I am refering to the flow below...
>>>>
>>>> The backup flow should be (all api calls are engine REST API):
>>> I disagree with this, by following this pattern you move internal
>>> logic (even more than this given implementation specific logic) to
>>> the api users instead of abstracting it and creating solid api for
>>> backups,
>>>
>>> +++++++++ several api calls for single backup ++++++++++++++
>>>
>>>
>>> cons:
>>> ====
>>>
>>> 1. user have to be aware of internal (implementation specific) flow of each provider in order to be able creating backup
>>>     1.1 call add(a)->do(a)->del(a)->do(b)->etc.
>>> 2. no flexibility for backup provider to change the backup flow (as it would be considered an api break)
>>> 3. no atomic resolution for the /backup failures
>>>     3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>>>          that user have to del(a) manually and if it's deeper in the flow
>>>          every api user will have to implement own rollbacks to the steps that took
>>>          place before the failure.
>>> 4. backward compatibility becomes an issue when exposing internals in such a low level.
>>> 5. new features expose becomes much complex task cause they may require performing extra
>>>     step/s in a middle.
>>>
>>> pros:
>>> ====
>>>
>>> can't think of any ..., cause it doesn't give you flexibility, but creates unnatural
>>> (to public APIs) complexity.
>> I was able to dig the below from one of the older mails. I had exactly this Q on why we need separate APIs Vs 1 and this is what Ayal had to say
>>
>> Deepak asked:
>>
>> /Wouldn't it be better if there was just 1 REST API for startign backup
>> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
>> internally caused the above individual APIs to get called ?
>> Why do we have to expose each of the steps in backup process as
>> individual REST APIs ?
>> /
>>
>> Ayal said:
>>
>> /1. Because a single API assumes that I always want to backup everything which isn't necessarily true (normally backup policy for system disk and data disks are different)
> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
>
>> 2. Going forward we will have disk level snapshot in which case the backup API would have to change.
> same
>
>> 3. You could always return the vm config per disk when calling "prepare backup disk" which would be a bit redundant now but once we have disk level snapshots it would be more relevant and this way remove 1 API call now.
>> Later on when we do have disk level snaps it could be an option to the command to take the snap or something I guess.
> it's only proves my point, we can/should hide this under api abstraction
> rather than forcing users using different flows now and then.
>
>> /
>> So i kind of agreed to what Ayal said... from the perspective that it provides flexibility to the end user using the API, as to how he/she wants to script/use it.
>> What do you think ?  I just wanted to put the above here... so that we are sure we can considering all aspects before making the decision on 1 Vs many APIs.
> i've mentioned this before, - i don't think this is about flexibility, if you want to allow
> user doing several things differently, - expose several api for same thing (like method overloads),
> but it doesn't mean that we should be exposing implementation internals to end user.
>
>> 1 API provides simplicity, but getting flexibility using 1 API would mean passing addnl params.. for eg> If i want to backup a particular vmdisk only, then the xml below would change, rite ?
> no, you misunderstood, see url [1] from my previous eamil, you already doing this for specific disk - yyy
>
> [1] /api/vms/xxx/disks/yyy/backups

I see your point now.... thanks for correcting. I think i am ok with 1 
API approach then.


>> Questions...
>>
>> 1) My current implmn of prepareBackupDisk in VDSM (pls see http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>> returns drive and cbtinfo as dicts back to engine. The understnading here is that the drive returned will be a dict which has all the necessary info
>> to represent a NBD drive to libvirt... hence this drive will be passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>
>> Note that i cannot return a disk UUID (like VDSM does for create volume case).. because in preparebackupdisk case the image is exported usign qemu-nbd
>> as a block device over the network... and hence its not a regular vdsm-type disk, hence no UUID, Agree ?
>>
>> 2) In "attach disk to VM" REST API user will pass the drive dict which he/she got as part of preparebackupdisk api.. as-is
>> and VDSM will need to add support for accepting NBD disk as a valid disktype in VM.hotplugDisk() code of VDSM.
>>>> The ability to add a network disk is present in libvirtvm.py... as part of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
>>>> but it doesn't work at the API.py level.. its only from prepareImage onwards... hence the need to modify API.py to accept NBD disk type and connect
>>>> to the libvirtvm.py appropriately
>>>>
>>>> Is this acceptable.. otherwise there is no good way to pass the NBD drive's info back to VDSM as part of existing "attach disk to VM" API
>>>> Also does the existing "attach disk to VM" API work if a pre-loaded drive dict/hashmap is provided by the user. This will have drive->type = network
>>>> and not file/blcok as is currently supported
>> Mike,
>>      Could you provide your views on this pls ? How do we represent the NBD disk and pass it back to VDSM as part of 'hotplugDisk' API
>> Currently hotplugDisk doesn't support drive of type = network!


What do you think of the above ? I am guessing your overlooked this in 
the prev mail ?
I believe my Q above is still relevant irrespective of 1 API or multiple 
API.



From liumbj at linux.vnet.ibm.com  Wed Jun  5 14:27:41 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Wed, 05 Jun 2013 22:27:41 +0800
Subject: Is there a way to detect congestion in ovirt storage domains
Message-ID: <51AF4ADD.6090706@linux.vnet.ibm.com>

Hi all,

I found that its a little difficult to detect io bandwidth congestion in 
ovirt storage domains supported by NFS or GlusterFs.

For block based storage, it's easier to detect, since you can use some 
tool like iostat.For the case of file system based storage, it's much 
harder.

Does anybody get an idea or have experience on this? Thanks in advance.

Best regards,
Mei Liu(Rose)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130605/ae850ddd/attachment.html>

From dfediuck at redhat.com  Wed Jun  5 14:26:55 2013
From: dfediuck at redhat.com (Doron Fediuck)
Date: Wed, 5 Jun 2013 10:26:55 -0400 (EDT)
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <51AF0F87.2000809@linux.vnet.ibm.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
	<26148712.4071809.1370423021002.JavaMail.root@redhat.com>
	<51AF0F87.2000809@linux.vnet.ibm.com>
Message-ID: <666894327.19206560.1370442415368.JavaMail.root@redhat.com>

Hi Mei Liu,
sorry for the delay.

I agree that the advanced approach is getting complicated, and
that we can allow mom to control parts of it.
Having that said, I'd still want you to look at the bigger picture
of a VM QoS.
Please refer to the discussion we had on network QoS, where we
managed to define it in a way which going forward will be more
user friendly on the one hand, but can be translated into mom-policy
on the other hand.
See: http://lists.ovirt.org/pipermail/engine-devel/2013-June/004764.html

As you can see there, we defined a network profile, which includes a QoS
element. Going forward this element will be taken from a policy, which
will hold additional QoS aspects such as I/O, CPU and memory. In addition
to that, the policy may include a reference to a quota allocation as well
as other SLA related properties such as HA.

So in this light, I'd like to see a disk-profile, which holds the QoS element
and possibly other storage related properties, such as mom-auto-tune arguments.
So once an admin sets this profile, the VM creator (or disk creator) can
choose the relevant profile he needs for the disk and assign it to the disk.
Once we have this in place, the engine will provide this policy to mom on
VM creation, and mom will enforce I/O (in this case, but also network and
others going forward) on the VM based on the policy it got. If the policy
allows I/O auto-tuning, then mom will manage it for that disk.

The benefits to this approach are that it's more holistic from system
perspective. It should allow easy management and simple defaults which
users should not deal with unless they have the relevant permissions such
as disk creators.

So in this context, I'd love to see a definition of disk or image profile,
and the storage QoS element it holds. The VDSM API should already support
policy updates for mom, so we only need to introduce the libvirt API elements.

Feedback is more than welcomed.
Doron

----- Original Message -----
> From: "Mei Liu" <liumbj at linux.vnet.ibm.com>
> To: "Allon Mureinik" <amureini at redhat.com>
> Cc: "Doron Fediuck" <dfediuck at redhat.com>, arch at ovirt.org
> Sent: Wednesday, June 5, 2013 1:14:31 PM
> Subject: Re: SLA feature for storage I/O bandwidth
> 
> On 06/05/2013 05:03 PM, Allon Mureinik wrote:
> > Hi Mei,
> >
> > How are you treating shared disks?
> > Is the limitation defined per disk (as a total), or per disk-vm relation?
> >
> The existing API in libvirt is to limit the  per vm's vDisk. e.g. hda of VM1
> 
> So the limitation is on per vm's vDisk, if we don't consider quota for
> SD in the design (the basic design). That limitation value is adjusted
> based on if backend storage(localfs, nfs, glusterfs...) is heavily used.
> 
> SD's IO bandwidth quota is to constrain the sum of vDisk limitation's
> lower bound(min value of limitation, it is similar to the concept of
> reserving). These vDisks use the volumes in the SD.
> 
> 
> > ------------------------------------------------------------------------
> >
> >     *From: *"Mei Liu" <liumbj at linux.vnet.ibm.com>
> >     *To: *"Doron Fediuck" <dfediuck at redhat.com>
> >     *Cc: *arch at ovirt.org
> >     *Sent: *Wednesday, June 5, 2013 11:11:34 AM
> >     *Subject: *Re: SLA feature for storage I/O bandwidth
> >
> >     Hi Doron,
> >     After the discussion, we found that the last version of design is
> >     a little complex, so I simplified it and post it to
> >
> >     http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth
> >
> >     We want to let MOM in each host decides how to tune the bandwidth
> >     limit of vms in that host instead of letting engine to make the
> >     whole decision based on statistics from vms in diverse hosts.
> >     Maybe we can consider starting from the basic one in wiki.
> >
> >     Thanks & best regards,
> >     Mei Liu(Rose)
> >
> >     -------- Original Message --------
> >     Subject: 	Re: SLA feature for storage I/O bandwidth
> >     Date: 	Mon, 03 Jun 2013 18:28:46 +0800
> >     From: 	Mei Liu <liumbj at linux.vnet.ibm.com>
> >     To: 	Doron Fediuck <dfediuck at redhat.com>
> >     CC: 	arch at ovirt.org
> >
> >
> >
> >     On 05/29/2013 11:34 PM, Doron Fediuck wrote:
> >     > ----- Original Message -----
> >     >> From: "Mei Liu"<liumbj at linux.vnet.ibm.com>
> >     >> To: "Dave Neary"<dneary at redhat.com>
> >     >> Cc:arch at ovirt.org
> >     >> Sent: Wednesday, May 29, 2013 11:35:12 AM
> >     >> Subject: Re: SLA feature for storage I/O bandwidth
> >     >>
> >     >> On 05/29/2013 03:42 PM, Dave Neary wrote:
> >     >>> Hi Mei Liu,
> >     >>>
> >     >>> On 05/28/2013 10:18 AM, Mei Liu wrote:
> >     >>>> I created a drafted wiki page on  design of storage I/O bandwidth
> >     >>>> SLA in
> >     >>>> the following link:
> >     >>>>
> >     >>>>http://www.ovirt.org/SLA_for_storage_resource  .
> >     >>>>
> >     >>>> I will appreciate the efforts if anyone who works on ovirt engine,
> >     >>>> vdsm
> >     >>>> or mom could give some comments. TIA.
> >     >>> Just out of interest - which version of oVirt are you targeting
> >     >>> this
> >     >>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
> >     >>> feature freeze (but we have a release meeting later to discuss
> >     >>> that).
> >     >>>
> >     >>> Thanks,
> >     >>> Dave.
> >     >>>
> >     >> Hi Dave,
> >     >> The basic i/o tune functionality for vdsm is almost ready. However,
> >     >> there is nothing written on the Engine side and no policy for
> >     >> automatic
> >     >> tuning is applied yet.
> >     >> I am not sure if the basic functionality can target 3.3.
> >     >>
> >     >>
> >     >> Best regards,
> >     >> Mei Liu
> >     >>
> >     > Hi Mey Liu,
> >     > I'm still going over the wiki, but a few things we need to consider;
> >     > First of all QoS for storage I/O bandwidth is a part of a larger SLA
> >     > policy which may include network QoS, CPU and memory QoS and the
> >     > quota
> >     > we implement today.
> >     >
> >     > So first of all we need to make sure your design does not conflict
> >     > the other QoS parts, which is what I'm looking into now.
> >     >
> >     > Additionally, using the quota term is confusing as oVirt already has
> >     > quota today, and cpu API has his own quota definition. So please try
> >     > to come up with a different terminology.
> >     >
> >     > I like your idea of setting an initial value but I need some more
> >     > time
> >     > for it to come up with my insights.
> >     > Also, I completely agree with your concept of letting mom handle
> >     > it in host level. We need to verify it does not break anything
> >     > related
> >     > to SPM. This is something we need to synchronize with the storage
> >     > guys.
> >     >
> >     > Looking into the engine area, we should start thinking on how this
> >     > will
> >     > be supported in the main storage entities and VM / template /
> >     > instance
> >     > entities. So you may want to add a section on this as well. This
> >     > leads
> >     > me to think of importing and exporting a VM which may want to
> >     > maintain
> >     > the defined IO QoS. Any thoughts around it?
> >     >
> >     > Doron
> >     >
> >     Hi Doron,
> >     Thanks for your questions and insightful thoughts. They are really
> >     inspiring.
> >
> >     I updated the design in
> >     http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth  .
> >     This time, I add  storage I/O bandwidth control according to the quota
> >     design and the SLA tries to ensure the reserved bandwidth for vDisks.
> >     It requires the engine  to make the tuning decision, since vm uses the
> >     SD volumes may  reside on different hosts and only engine can obtain
> >     the
> >     global information.
> >     I think this design will not lead to problem when importing or
> >     exporting
> >     a VM.
> >
> >     I will appreciate if you can take a look at the new design.
> >
> >     Best regards,
> >     Mei Liu (Rose)
> >
> >     _______________________________________________
> >     Arch mailing list
> >     Arch at ovirt.org
> >     http://lists.ovirt.org/mailman/listinfo/arch
> >
> >
> >
> >
> >
> >     _______________________________________________
> >     Arch mailing list
> >     Arch at ovirt.org
> >     http://lists.ovirt.org/mailman/listinfo/arch
> >
> >
> 
> 


From snmishra at linux.vnet.ibm.com  Wed Jun  5 16:16:44 2013
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 05 Jun 2013 09:16:44 -0700
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF2878.1080500@linux.vnet.ibm.com>
References: <51A76B36.7030408@linux.vnet.ibm.com>
	<51AF12B2.3000903@redhat.com> <51AF1DA2.5070305@linux.vnet.ibm.com>
	<51AF257D.6040800@redhat.com> <51AF2878.1080500@linux.vnet.ibm.com>
Message-ID: <20130605091644.Horde.PKK8pJir309Rr2RslXZnOtA@imap.linux.ibm.com>


Quoting Deepak C Shetty <deepakcs at linux.vnet.ibm.com>:

> On 06/05/2013 05:18 PM, Michael Pasternak wrote:
>> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
>>> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
>>>> Hi Deepak,
>>>>
>>>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>>>>> Hi All,
>>>>>     Per the previous mail from Ayal.. I am refering to the flow below...
>>>>>
>>>>> The backup flow should be (all api calls are engine REST API):
>>>> I disagree with this, by following this pattern you move internal
>>>> logic (even more than this given implementation specific logic) to
>>>> the api users instead of abstracting it and creating solid api for
>>>> backups,
>>>>
>>>> +++++++++ several api calls for single backup ++++++++++++++
>>>>
>>>>
>>>> cons:
>>>> ====
>>>>
>>>> 1. user have to be aware of internal (implementation specific)  
>>>> flow of each provider in order to be able creating backup
>>>>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
>>>> 2. no flexibility for backup provider to change the backup flow  
>>>> (as it would be considered an api break)
>>>> 3. no atomic resolution for the /backup failures
>>>>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>>>>         that user have to del(a) manually and if it's deeper in the flow
>>>>         every api user will have to implement own rollbacks to  
>>>> the steps that took
>>>>         place before the failure.
>>>> 4. backward compatibility becomes an issue when exposing  
>>>> internals in such a low level.
>>>> 5. new features expose becomes much complex task cause they may  
>>>> require performing extra
>>>>    step/s in a middle.
>>>>
>>>> pros:
>>>> ====
>>>>
>>>> can't think of any ..., cause it doesn't give you flexibility,  
>>>> but creates unnatural
>>>> (to public APIs) complexity.
>>> I was able to dig the below from one of the older mails. I had  
>>> exactly this Q on why we need separate APIs Vs 1 and this is what  
>>> Ayal had to say
>>>
>>> Deepak asked:
>>>
>>> /Wouldn't it be better if there was just 1 REST API for startign backup
>>> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
>>> internally caused the above individual APIs to get called ?
>>> Why do we have to expose each of the steps in backup process as
>>> individual REST APIs ?
>>> /
>>>
>>> Ayal said:
>>>
>>> /1. Because a single API assumes that I always want to backup  
>>> everything which isn't necessarily true (normally backup policy  
>>> for system disk and data disks are different)
>> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
>>
>>> 2. Going forward we will have disk level snapshot in which case  
>>> the backup API would have to change.
>> same
>>
>>> 3. You could always return the vm config per disk when calling  
>>> "prepare backup disk" which would be a bit redundant now but once  
>>> we have disk level snapshots it would be more relevant and this  
>>> way remove 1 API call now.
>>> Later on when we do have disk level snaps it could be an option to  
>>> the command to take the snap or something I guess.
>> it's only proves my point, we can/should hide this under api abstraction
>> rather than forcing users using different flows now and then.
>>
>>> /
>>> So i kind of agreed to what Ayal said... from the perspective that  
>>> it provides flexibility to the end user using the API, as to how  
>>> he/she wants to script/use it.
>>> What do you think ?  I just wanted to put the above here... so  
>>> that we are sure we can considering all aspects before making the  
>>> decision on 1 Vs many APIs.
>> i've mentioned this before, - i don't think this is about  
>> flexibility, if you want to allow
>> user doing several things differently, - expose several api for  
>> same thing (like method overloads),
>> but it doesn't mean that we should be exposing implementation  
>> internals to end user.
>>
>>> 1 API provides simplicity, but getting flexibility using 1 API  
>>> would mean passing addnl params.. for eg> If i want to backup a  
>>> particular vmdisk only, then the xml below would change, rite ?
>> no, you misunderstood, see url [1] from my previous eamil, you  
>> already doing this for specific disk - yyy
>>
>> [1] /api/vms/xxx/disks/yyy/backups
>
> I see your point now.... thanks for correcting. I think i am ok with  
> 1 API approach then.

Added Ayal to the CC. Since he is the owner of this feature. Would  
like to hear his opinion.

-Sharad

>
>
>>> Questions...
>>>
>>> 1) My current implmn of prepareBackupDisk in VDSM (pls see  
>>> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>>> returns drive and cbtinfo as dicts back to engine. The  
>>> understnading here is that the drive returned will be a dict which  
>>> has all the necessary info
>>> to represent a NBD drive to libvirt... hence this drive will be  
>>> passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>>
>>> Note that i cannot return a disk UUID (like VDSM does for create  
>>> volume case).. because in preparebackupdisk case the image is  
>>> exported usign qemu-nbd
>>> as a block device over the network... and hence its not a regular  
>>> vdsm-type disk, hence no UUID, Agree ?
>>>
>>> 2) In "attach disk to VM" REST API user will pass the drive dict  
>>> which he/she got as part of preparebackupdisk api.. as-is
>>> and VDSM will need to add support for accepting NBD disk as a  
>>> valid disktype in VM.hotplugDisk() code of VDSM.
>>>>> The ability to add a network disk is present in libvirtvm.py...  
>>>>> as part of my GlusterSD work, sicne gluster is represented as a  
>>>>> NBD to QEMU
>>>>> but it doesn't work at the API.py level.. its only from  
>>>>> prepareImage onwards... hence the need to modify API.py to  
>>>>> accept NBD disk type and connect
>>>>> to the libvirtvm.py appropriately
>>>>>
>>>>> Is this acceptable.. otherwise there is no good way to pass the  
>>>>> NBD drive's info back to VDSM as part of existing "attach disk  
>>>>> to VM" API
>>>>> Also does the existing "attach disk to VM" API work if a  
>>>>> pre-loaded drive dict/hashmap is provided by the user. This will  
>>>>> have drive->type = network
>>>>> and not file/blcok as is currently supported
>>> Mike,
>>>     Could you provide your views on this pls ? How do we represent  
>>> the NBD disk and pass it back to VDSM as part of 'hotplugDisk' API
>>> Currently hotplugDisk doesn't support drive of type = network!
>
>
> What do you think of the above ? I am guessing your overlooked this  
> in the prev mail ?
> I believe my Q above is still relevant irrespective of 1 API or multiple API.
>
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch





From snmishra at linux.vnet.ibm.com  Wed Jun  5 16:19:55 2013
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 05 Jun 2013 09:19:55 -0700
Subject: Fwd: Need response to emails and patches
Message-ID: <20130605091955.Horde.e1iF5Jir309Rr2UrExKXOxA@imap.linux.ibm.com>


Moving the discussion to arch ML.

-Sharad

From:	Deepak C Shetty <deepakcs at linux.vnet.ibm.com>
To:	Sharad Mishra/Beaverton/IBM at IBMUS,
Cc:	laravot <laravot%redhat.com._amureini at us.ibm.com>, Ayal Baron
             <abaron at redhat.com>, agl at linux.vnet.ibm.com, Sean Cohen
             <scohen at redhat.com>, Deepak C Shetty
             <deepakcs at linux.vnet.ibm.com>, Federico Simoncelli
             <fsimonce at redhat.com>
Date:	06/04/2013 09:35 PM
Subject:	Re: Need response to emails and patches



Sharad,
     I was actually planning to integrate with engine patches once the "add
disk to backup VM" and its counterpart were ready. One of the mails i sent
was to get clarity on whether we need a new API for "add disk to backup VM"
or not. I feel the existign hotplug disk may not work... since it only
cares abt file and block type drives.. while qemu-nbd is a network type
drive and i don't see engine support for sending drive params for a NBD
drive

Fede,
     I know you are busy, can you atleast help provide your views on the
mail I sent with regards to "add disk to backup VM"... we need to get
consensus on whether we need a new API and if not. what changes will happen
in engine and VDSM to make the existing API work with NBD drive

thanx,
deepak

On 06/04/2013 09:48 PM, Sharad Mishra wrote:


       We also need to decide on whether we use single API call for
       backup/restore or multiple APIs as documented in feature wiki.

       I have posted patches on gerrit that follow the wiki design. Haven't
       had any reviews yet.

       Liron, Let me know if you have any question on restore work. Also,
       please take a look at the engine patches for backup.

       Deepak,  Is your patch at a point where we can integrate with engine
       and see if the handshake between engine API to prepare and teardown
       occur properly?

       Regards,
       Sharad Mishra
       Open Virtualization
       Linux Technology Center
       IBM

       Inactive hide details for Deepak C Shetty ---06/04/2013
       07:17:33 AM---Hi Sean, I sent few mails with subject...Deepak C
       Shetty ---06/04/2013 07:17:33 AM---Hi Sean,      I sent few mails
       with subject...

       From: Deepak C Shetty <deepakcs at linux.vnet.ibm.com>
       To: Sean Cohen <scohen at redhat.com>,
       Cc: Ayal Baron <abaron at redhat.com>, Sharad
       Mishra/Beaverton/IBM at IBMUS, agl at linux.vnet.ibm.com, Deepak C Shetty
       <deepakcs at linux.vnet.ibm.com>
       Date: 06/04/13 07:17 AM
       Subject: Need response to emails and patches





       Hi Sean,
           I sent few mails with subject...

       1) Follow-up on the Storage Array offload meeting
       2) Few questions on Backup Restore API implmn.

       last week.. would appreciate if we can get some responses on the
       same,
       which will help in working further.

       Also i have sent my next set of patch(es) on backup/restore after
       implementing fede and ayal's review comments @

       http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master
       +topic:backup-restore,n,z

       Can you help schedule a call for doing the code review ?

       thanx,
       deepak

----- End forwarded message -----



From snmishra at linux.vnet.ibm.com  Wed Jun  5 16:21:46 2013
From: snmishra at linux.vnet.ibm.com (snmishra at linux.vnet.ibm.com)
Date: Wed, 05 Jun 2013 09:21:46 -0700
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <20130605091644.Horde.PKK8pJir309Rr2RslXZnOtA@imap.linux.ibm.com>
References: <51A76B36.7030408@linux.vnet.ibm.com>
	<51AF12B2.3000903@redhat.com> <51AF1DA2.5070305@linux.vnet.ibm.com>
	<51AF257D.6040800@redhat.com> <51AF2878.1080500@linux.vnet.ibm.com>
	<20130605091644.Horde.PKK8pJir309Rr2RslXZnOtA@imap.linux.ibm.com>
Message-ID: <20130605092146.Horde.QtRSy5ir309Rr2WaVJr3OxA@imap.linux.ibm.com>


Quoting snmishra at linux.vnet.ibm.com:

> Quoting Deepak C Shetty <deepakcs at linux.vnet.ibm.com>:
>
>> On 06/05/2013 05:18 PM, Michael Pasternak wrote:
>>> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
>>>> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
>>>>> Hi Deepak,
>>>>>
>>>>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>>>>>> Hi All,
>>>>>>    Per the previous mail from Ayal.. I am refering to the flow below...
>>>>>>
>>>>>> The backup flow should be (all api calls are engine REST API):
>>>>> I disagree with this, by following this pattern you move internal
>>>>> logic (even more than this given implementation specific logic) to
>>>>> the api users instead of abstracting it and creating solid api for
>>>>> backups,
>>>>>
>>>>> +++++++++ several api calls for single backup ++++++++++++++
>>>>>
>>>>>
>>>>> cons:
>>>>> ====
>>>>>
>>>>> 1. user have to be aware of internal (implementation specific)  
>>>>> flow of each provider in order to be able creating backup
>>>>>   1.1 call add(a)->do(a)->del(a)->do(b)->etc.
>>>>> 2. no flexibility for backup provider to change the backup flow  
>>>>> (as it would be considered an api break)
>>>>> 3. no atomic resolution for the /backup failures
>>>>>   3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>>>>>        that user have to del(a) manually and if it's deeper in the flow
>>>>>        every api user will have to implement own rollbacks to  
>>>>> the steps that took
>>>>>        place before the failure.
>>>>> 4. backward compatibility becomes an issue when exposing  
>>>>> internals in such a low level.
>>>>> 5. new features expose becomes much complex task cause they may  
>>>>> require performing extra
>>>>>   step/s in a middle.
>>>>>
>>>>> pros:
>>>>> ====
>>>>>
>>>>> can't think of any ..., cause it doesn't give you flexibility,  
>>>>> but creates unnatural
>>>>> (to public APIs) complexity.
>>>> I was able to dig the below from one of the older mails. I had  
>>>> exactly this Q on why we need separate APIs Vs 1 and this is what  
>>>> Ayal had to say
>>>>
>>>> Deepak asked:
>>>>
>>>> /Wouldn't it be better if there was just 1 REST API for startign backup
>>>> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
>>>> internally caused the above individual APIs to get called ?
>>>> Why do we have to expose each of the steps in backup process as
>>>> individual REST APIs ?
>>>> /
>>>>
>>>> Ayal said:
>>>>
>>>> /1. Because a single API assumes that I always want to backup  
>>>> everything which isn't necessarily true (normally backup policy  
>>>> for system disk and data disks are different)
>>> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
>>>
>>>> 2. Going forward we will have disk level snapshot in which case  
>>>> the backup API would have to change.
>>> same
>>>
>>>> 3. You could always return the vm config per disk when calling  
>>>> "prepare backup disk" which would be a bit redundant now but once  
>>>> we have disk level snapshots it would be more relevant and this  
>>>> way remove 1 API call now.
>>>> Later on when we do have disk level snaps it could be an option  
>>>> to the command to take the snap or something I guess.
>>> it's only proves my point, we can/should hide this under api abstraction
>>> rather than forcing users using different flows now and then.
>>>
>>>> /
>>>> So i kind of agreed to what Ayal said... from the perspective  
>>>> that it provides flexibility to the end user using the API, as to  
>>>> how he/she wants to script/use it.
>>>> What do you think ?  I just wanted to put the above here... so  
>>>> that we are sure we can considering all aspects before making the  
>>>> decision on 1 Vs many APIs.
>>> i've mentioned this before, - i don't think this is about  
>>> flexibility, if you want to allow
>>> user doing several things differently, - expose several api for  
>>> same thing (like method overloads),
>>> but it doesn't mean that we should be exposing implementation  
>>> internals to end user.
>>>
>>>> 1 API provides simplicity, but getting flexibility using 1 API  
>>>> would mean passing addnl params.. for eg> If i want to backup a  
>>>> particular vmdisk only, then the xml below would change, rite ?
>>> no, you misunderstood, see url [1] from my previous eamil, you  
>>> already doing this for specific disk - yyy
>>>
>>> [1] /api/vms/xxx/disks/yyy/backups
>>
>> I see your point now.... thanks for correcting. I think i am ok  
>> with 1 API approach then.
>
> Added Ayal to the CC. Since he is the owner of this feature. Would  
> like to hear his opinion.
>
> -Sharad
>
>>
>>
>>>> Questions...
>>>>
>>>> 1) My current implmn of prepareBackupDisk in VDSM (pls see  
>>>> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>>>> returns drive and cbtinfo as dicts back to engine. The  
>>>> understnading here is that the drive returned will be a dict  
>>>> which has all the necessary info
>>>> to represent a NBD drive to libvirt... hence this drive will be  
>>>> passed as-is in the "attach disk to VM" REST API. Is this correct ?
>>>>
>>>> Note that i cannot return a disk UUID (like VDSM does for create  
>>>> volume case).. because in preparebackupdisk case the image is  
>>>> exported usign qemu-nbd
>>>> as a block device over the network... and hence its not a regular  
>>>> vdsm-type disk, hence no UUID, Agree ?
>>>>
>>>> 2) In "attach disk to VM" REST API user will pass the drive dict  
>>>> which he/she got as part of preparebackupdisk api.. as-is
>>>> and VDSM will need to add support for accepting NBD disk as a  
>>>> valid disktype in VM.hotplugDisk() code of VDSM.
>>>>>> The ability to add a network disk is present in libvirtvm.py...  
>>>>>> as part of my GlusterSD work, sicne gluster is represented as a  
>>>>>> NBD to QEMU
>>>>>> but it doesn't work at the API.py level.. its only from  
>>>>>> prepareImage onwards... hence the need to modify API.py to  
>>>>>> accept NBD disk type and connect
>>>>>> to the libvirtvm.py appropriately
>>>>>>
>>>>>> Is this acceptable.. otherwise there is no good way to pass the  
>>>>>> NBD drive's info back to VDSM as part of existing "attach disk  
>>>>>> to VM" API
>>>>>> Also does the existing "attach disk to VM" API work if a  
>>>>>> pre-loaded drive dict/hashmap is provided by the user. This  
>>>>>> will have drive->type = network
>>>>>> and not file/blcok as is currently supported
>>>> Mike,
>>>>    Could you provide your views on this pls ? How do we represent  
>>>> the NBD disk and pass it back to VDSM as part of 'hotplugDisk' API
>>>> Currently hotplugDisk doesn't support drive of type = network!
>>
>>
>> What do you think of the above ? I am guessing your overlooked this  
>> in the prev mail ?
>> I believe my Q above is still relevant irrespective of 1 API or  
>> multiple API.
>>
>> _______________________________________________
>> Arch mailing list
>> Arch at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/arch





From iheim at redhat.com  Wed Jun  5 18:45:43 2013
From: iheim at redhat.com (Itamar Heim)
Date: Wed, 05 Jun 2013 21:45:43 +0300
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <666894327.19206560.1370442415368.JavaMail.root@redhat.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
	<26148712.4071809.1370423021002.JavaMail.root@redhat.com>
	<51AF0F87.2000809@linux.vnet.ibm.com>
	<666894327.19206560.1370442415368.JavaMail.root@redhat.com>
Message-ID: <51AF8757.1020300@redhat.com>

On 06/05/2013 05:26 PM, Doron Fediuck wrote:
> Hi Mei Liu,
> sorry for the delay.
>
> I agree that the advanced approach is getting complicated, and
> that we can allow mom to control parts of it.
> Having that said, I'd still want you to look at the bigger picture
> of a VM QoS.
> Please refer to the discussion we had on network QoS, where we
> managed to define it in a way which going forward will be more
> user friendly on the one hand, but can be translated into mom-policy
> on the other hand.
> See: http://lists.ovirt.org/pipermail/engine-devel/2013-June/004764.html
>
> As you can see there, we defined a network profile, which includes a QoS
> element. Going forward this element will be taken from a policy, which
> will hold additional QoS aspects such as I/O, CPU and memory. In addition
> to that, the policy may include a reference to a quota allocation as well
> as other SLA related properties such as HA.
>
> So in this light, I'd like to see a disk-profile, which holds the QoS element
> and possibly other storage related properties, such as mom-auto-tune arguments.
> So once an admin sets this profile, the VM creator (or disk creator) can
> choose the relevant profile he needs for the disk and assign it to the disk.
> Once we have this in place, the engine will provide this policy to mom on
> VM creation, and mom will enforce I/O (in this case, but also network and
> others going forward) on the VM based on the policy it got. If the policy
> allows I/O auto-tuning, then mom will manage it for that disk.
>
> The benefits to this approach are that it's more holistic from system
> perspective. It should allow easy management and simple defaults which
> users should not deal with unless they have the relevant permissions such
> as disk creators.
>
> So in this context, I'd love to see a definition of disk or image profile,
> and the storage QoS element it holds. The VDSM API should already support
> policy updates for mom, so we only need to introduce the libvirt API elements.
>
> Feedback is more than welcomed.

I like the concept of using disk profile similar to network profiles.
how were you envisioning managing these entities for disks?
for networks, iiuc, the profile are defined per network, then 
permissions are defined on them (which users can use each profile, per 
network).
where would disk profiles be defined? storage domain? system level? etc.



From dfediuck at redhat.com  Thu Jun  6 10:50:32 2013
From: dfediuck at redhat.com (Doron Fediuck)
Date: Thu, 6 Jun 2013 06:50:32 -0400 (EDT)
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <51AF8757.1020300@redhat.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
	<26148712.4071809.1370423021002.JavaMail.root@redhat.com>
	<51AF0F87.2000809@linux.vnet.ibm.com>
	<666894327.19206560.1370442415368.JavaMail.root@redhat.com>
	<51AF8757.1020300@redhat.com>
Message-ID: <343088563.19807089.1370515832227.JavaMail.root@redhat.com>

----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Doron Fediuck" 
> Cc: "Mei Liu" <liumbj at linux.vnet.ibm.com>, arch at ovirt.org
> Sent: Wednesday, June 5, 2013 9:45:43 PM
> Subject: Re: SLA feature for storage I/O bandwidth
> 
> On 06/05/2013 05:26 PM, Doron Fediuck wrote:
> > Hi Mei Liu,
> > sorry for the delay.
> >
> > I agree that the advanced approach is getting complicated, and
> > that we can allow mom to control parts of it.
> > Having that said, I'd still want you to look at the bigger picture
> > of a VM QoS.
> > Please refer to the discussion we had on network QoS, where we
> > managed to define it in a way which going forward will be more
> > user friendly on the one hand, but can be translated into mom-policy
> > on the other hand.
> > See: http://lists.ovirt.org/pipermail/engine-devel/2013-June/004764.html
> >
> > As you can see there, we defined a network profile, which includes a QoS
> > element. Going forward this element will be taken from a policy, which
> > will hold additional QoS aspects such as I/O, CPU and memory. In addition
> > to that, the policy may include a reference to a quota allocation as well
> > as other SLA related properties such as HA.
> >
> > So in this light, I'd like to see a disk-profile, which holds the QoS
> > element
> > and possibly other storage related properties, such as mom-auto-tune
> > arguments.
> > So once an admin sets this profile, the VM creator (or disk creator) can
> > choose the relevant profile he needs for the disk and assign it to the
> > disk.
> > Once we have this in place, the engine will provide this policy to mom on
> > VM creation, and mom will enforce I/O (in this case, but also network and
> > others going forward) on the VM based on the policy it got. If the policy
> > allows I/O auto-tuning, then mom will manage it for that disk.
> >
> > The benefits to this approach are that it's more holistic from system
> > perspective. It should allow easy management and simple defaults which
> > users should not deal with unless they have the relevant permissions such
> > as disk creators.
> >
> > So in this context, I'd love to see a definition of disk or image profile,
> > and the storage QoS element it holds. The VDSM API should already support
> > policy updates for mom, so we only need to introduce the libvirt API
> > elements.
> >
> > Feedback is more than welcomed.
> 
> I like the concept of using disk profile similar to network profiles.
> how were you envisioning managing these entities for disks?
> for networks, iiuc, the profile are defined per network, then
> permissions are defined on them (which users can use each profile, per
> network).
> where would disk profiles be defined? storage domain? system level? etc.
> 

Network profile contains information on the connection of the VM to the
network, which is the vNIC. So using this concept, the disk is the
connection of a storage domain to a VM, thus disk profile holds the
information on this connection. So in this view permissions would be kept
at storage domain level, and we'll need to handle the profile when moving
(cloning?) a disk between SD's.


From dcaroest at redhat.com  Thu Jun  6 17:23:06 2013
From: dcaroest at redhat.com (David Caro)
Date: Thu, 06 Jun 2013 19:23:06 +0200
Subject: Gerrit connection issues
Message-ID: <51B0C57A.6080204@redhat.com>

Hi everyone!

We made some changes on the xinetd service to allow more connections to
avoid the connection refused problem when accessing the git repos. If
you still have problem or see errors like this one [1] on jenkins,
notify me so we can react and troubleshoot them.


Thanks!

[1]
http://jenkins.ovirt.org/view/vdsm/job/vdsm_unit_tests_gerrit/2711/console

-- 
David Caro

Red Hat Czech s.r.o.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro at redhat.com
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purky?ova 99/71, 612 45, Brno, Czech Republic
RHT Global #: 82-62605

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130606/453d2ec4/attachment.sig>

From abaron at redhat.com  Thu Jun  6 23:10:04 2013
From: abaron at redhat.com (Ayal Baron)
Date: Thu, 6 Jun 2013 19:10:04 -0400 (EDT)
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51AF257D.6040800@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
Message-ID: <350995831.12892634.1370560204679.JavaMail.root@redhat.com>



----- Original Message -----
> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
> > On 06/05/2013 03:58 PM, Michael Pasternak wrote:
> >> Hi Deepak,
> >>
> >> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
> >>> Hi All,
> >>>     Per the previous mail from Ayal.. I am refering to the flow below...
> >>>
> >>> The backup flow should be (all api calls are engine REST API):
> >> I disagree with this, by following this pattern you move internal
> >> logic (even more than this given implementation specific logic) to
> >> the api users instead of abstracting it and creating solid api for
> >> backups,

This is not internal logic.
I think what is not clear here is that we're *not* modelling a backup flow.
We're modelling a flow to allow a user to expose an existing disk to a virt-app / physical appliance, while the VM which owns the disk is still running.
There can be a horde of uses for this, off the top of my head:
1. obviously backup
2. scan disk for viruses externally
3. scan disk for inventory management
There will be many more which users come up with.

Designing the API around the specific usage (policy) is limiting and will lead to a lot more API calls due to the need to add new ones for every new usage someone thinks of. 

The API should provide users with useful mechanisms.
In this case what we need is the ability to get a point in time of a disk available for read (and in some cases write) on one of the hosts (and later on possibly over the net as well, assuming we secure it).
I do not always want to take a snapshot for example, I want to be able to use the API to expose an existing snapshot.
Once we add live merge, user may *choose* to merge immediately after backup is finished or to keep the snapshot (policy).




> >>
> >> +++++++++ several api calls for single backup ++++++++++++++
> >>
> >>
> >> cons:
> >> ====
> >>
> >> 1. user have to be aware of internal (implementation specific) flow of
> >> each provider in order to be able creating backup
> >>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
> >> 2. no flexibility for backup provider to change the backup flow (as it
> >> would be considered an api break)
> >> 3. no atomic resolution for the /backup failures
> >>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
> >>         that user have to del(a) manually and if it's deeper in the flow
> >>         every api user will have to implement own rollbacks to the steps
> >>         that took
> >>         place before the failure.
> >> 4. backward compatibility becomes an issue when exposing internals in such
> >> a low level.
> >> 5. new features expose becomes much complex task cause they may require
> >> performing extra
> >>    step/s in a middle.
> >>
> >> pros:
> >> ====
> >>
> >> can't think of any ..., cause it doesn't give you flexibility, but creates
> >> unnatural
> >> (to public APIs) complexity.
> > 
> > I was able to dig the below from one of the older mails. I had exactly this
> > Q on why we need separate APIs Vs 1 and this is what Ayal had to say
> > 
> > Deepak asked:
> > 
> > /Wouldn't it be better if there was just 1 REST API for startign backup
> > which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
> > internally caused the above individual APIs to get called ?
> > Why do we have to expose each of the steps in backup process as
> > individual REST APIs ?
> > /
> > 
> > Ayal said:
> > 
> > /1. Because a single API assumes that I always want to backup everything
> > which isn't necessarily true (normally backup policy for system disk and
> > data disks are different)
> 
> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
> 
> > 2. Going forward we will have disk level snapshot in which case the backup
> > API would have to change.
> 
> same
> 
> > 3. You could always return the vm config per disk when calling "prepare
> > backup disk" which would be a bit redundant now but once we have disk
> > level snapshots it would be more relevant and this way remove 1 API call
> > now.
> > Later on when we do have disk level snaps it could be an option to the
> > command to take the snap or something I guess.
> 
> it's only proves my point, we can/should hide this under api abstraction
> rather than forcing users using different flows now and then.
> 
> > /
> > So i kind of agreed to what Ayal said... from the perspective that it
> > provides flexibility to the end user using the API, as to how he/she wants
> > to script/use it.
> > What do you think ?  I just wanted to put the above here... so that we are
> > sure we can considering all aspects before making the decision on 1 Vs
> > many APIs.
> 
> i've mentioned this before, - i don't think this is about flexibility, if you
> want to allow
> user doing several things differently, - expose several api for same thing
> (like method overloads),
> but it doesn't mean that we should be exposing implementation internals to
> end user.
> 
> > 
> > 1 API provides simplicity, but getting flexibility using 1 API would mean
> > passing addnl params.. for eg> If i want to backup a particular vmdisk
> > only, then the xml below would change, rite ?
> 
> no, you misunderstood, see url [1] from my previous eamil, you already doing
> this for specific disk - yyy
> 
> [1] /api/vms/xxx/disks/yyy/backups
> 
> > But looks like either way, its possible to get things done...
> > I personally don't have much experience in the area of REST APIs to take
> > the call on whether we need 1 or multiple APIs... so I am still
> > inconclusive here (sorry about that!)
> > All i can say is that with 1 API approach too, we can get flexibility.
> > 
> > 
> >>
> >>
> >> +++++++++ single api call for single backup ++++++++++++++
> >>
> >> cons:
> >> ====
> >>
> >> nothing comes to my mind, are there any?
> >>
> >> pros:
> >> ====
> >>
> >> 1. user is not aware of any backup internals and can consume different
> >> /backup
> >>    providers using very same api and supplying parameters relevant for the
> >>    provider
> >>    he'd like to use.
> >> 2. backup providers can change/extend internal flow as much as they
> >> want/need, only
> >>    by modifying internal engine driver, user using abstraction over the
> >>    backup api
> >>    won't feel the difference.
> >> 3. backup providers can execute backup as single atomic operation and take
> >> care
> >>    for rollback cleanups in the transaction.
> >> 4. backup providers can easily maintain backward compatibility over
> >> abstraction
> >> 5. adding new features that require flow extension can be easily hidden
> >> under
> >>    the abstraction layer.
> >>
> >> also this is a common practice to expose abstraction layer to public,
> >> while
> >> vendors only have to implement the driver in order to be supported.
> >>
> >>
> >>> 1. create vm snapshot (the api call exists today)
> >>> 2. get VM Config (new API)
> >>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
> >>> return: paths to device on host as well as map of changed blocks) -
> >>> this should be called for every disk that needs to be backed up. Note
> >>> that VM snapshot takes a snap of *all* disks of the VM
> >>> 4. attach disk to backup vm (the api call exists today.  This assumes
> >>> virt app) - also has to be called per disk to back up
> >>> 5. At this point backup app would do the backup
> >>>
> >> this can be easily done under the single api call:
> >> =================================================
> >>
> >> POST /api/vms/xxx/disks/yyy/backups
> >>
> >> <backup>
> >>   <host id=aaa> #3
> >>   <backup_vm id=bbb> #1, #2
> >>   <delete_snapshot>true|false<delete_snapshot/> #7
> >> <backup/>
> >>
> >> 1. create vm snapshot => you already in context of vm, trigger snapshot on
> >> it
> >> 2. get VM Config => you already in context of vm, collect relevant meta
> >> 3. prepare backup disk => you in context of the disk already, do whatever
> >> is needed
> >> 4. attach disk to backup vm => you have backup_vm id in the request body
> >> 5. do the actual backup
> >>
> >> does this makes sense?
> >>
> >>> 5) detach disk (symmetric to 4)
> >>> 6. teardown backup disk (symmetric to 3)
> >>> 7. delete snap - This can only be called if VM is down today and is
> >>> not mandatory in any event.  Once we have live merge, it should be
> >>> policy driven (user should be able to choose whether to keep or delete)
> >>>
> >>> Questions...
> >>>
> >>> 1) My current implmn of prepareBackupDisk in VDSM (pls see
> >>> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
> >>> returns drive and cbtinfo as dicts back to engine. The understnading here
> >>> is that the drive returned will be a dict which has all the necessary
> >>> info
> >>> to represent a NBD drive to libvirt... hence this drive will be passed
> >>> as-is in the "attach disk to VM" REST API. Is this correct ?
> >>>
> >>> Note that i cannot return a disk UUID (like VDSM does for create volume
> >>> case).. because in preparebackupdisk case the image is exported usign
> >>> qemu-nbd
> >>> as a block device over the network... and hence its not a regular
> >>> vdsm-type disk, hence no UUID, Agree ?
> >>>
> >>> 2) In "attach disk to VM" REST API user will pass the drive dict which
> >>> he/she got as part of preparebackupdisk api.. as-is
> >>> and VDSM will need to add support for accepting NBD disk as a valid
> >>> disktype in VM.hotplugDisk() code of VDSM.
> > 
> >>>
> >>> The ability to add a network disk is present in libvirtvm.py... as part
> >>> of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
> >>> but it doesn't work at the API.py level.. its only from prepareImage
> >>> onwards... hence the need to modify API.py to accept NBD disk type and
> >>> connect
> >>> to the libvirtvm.py appropriately
> >>>
> >>> Is this acceptable.. otherwise there is no good way to pass the NBD
> >>> drive's info back to VDSM as part of existing "attach disk to VM" API
> >>> Also does the existing "attach disk to VM" API work if a pre-loaded drive
> >>> dict/hashmap is provided by the user. This will have drive->type =
> >>> network
> >>> and not file/blcok as is currently supported
> > 
> > Mike,
> >     Could you provide your views on this pls ? How do we represent the NBD
> >     disk and pass it back to VDSM as part of 'hotplugDisk' API
> > Currently hotplugDisk doesn't support drive of type = network!
> > 
> >>>
> >>> 3) After "attach disk to backup VM" REST API step.. the understnading is
> >>> that some backup vendor specific API will be called to tell the
> >>> backup appln....
> >>>     -- Block device (attached to the VM) to be used as src for backup
> >>>     -- CBT info of this blcok device (which was recd. as part of
> >>>     prepareBackupDisk API)
> >>> Is this correct ?
> >>>
> >>> thanx,
> >>> deepak
> >>>
> >>
> > 
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From kwade at redhat.com  Fri Jun  7 02:13:54 2013
From: kwade at redhat.com (Karsten 'quaid' Wade)
Date: Thu, 06 Jun 2013 19:13:54 -0700
Subject: Outage window :: resources.ovirt.org / Mailman / etc. :: 2013-06-07
	03:00 - 11:00 UTC
In-Reply-To: <1752941703.18568.1370569336755.JavaMail.cfusion@mail2.linode.com>
References: <1752941703.18568.1370569336755.JavaMail.cfusion@mail2.linode.com>
Message-ID: <51B141E2.3000707@redhat.com>

There is going to be an outage of resources.ovirt.org for up to 2 hours
sometime during an 8 hour service window.

The outage will occur 2013-06-07 between 03:00 and 11:00 UTC. To view in
your local time:

date -d '2013-06-07 03:00 UTC'
date -d '2013-06-07 11:00 UTC'

== Details ==

From the Linode service bulletin/reminder:

"This is just a friendly reminder that your maintenance window starts at
8PM PDT tonight and lasts until 4AM PDT.

Downtime from this maintenance is expected to be less than 120 minutes,
however please note that the entire maintenance window may be used if
required. Your Linode will be gracefully powered down and rebooted
during the maintenance. Services not configured to start on a reboot
will need to be manually started.

Additional information regarding this maintenance event can be found
here: <https://blog.linode.com/2013/05/17/fremont-upgrades/> "

== Affected services ==

* resources.ovirt.org
* Mailman (lists.ovirt.org)
* some backup services

=== Not-affected services ==

* www.ovirt.org / wiki.ovirt.org
* gerrit.ovirt.org
* jenkins.ovirt.org
* Anything at AlterWay and RackSpace facilities
** Foreman
** Puppet
** Jenkins slaves
** ...

== Future plans ==

We'll be migrating all services from this host and decommissioning it ASAP.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 255 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130606/a0e1b747/attachment.sig>

From liumbj at linux.vnet.ibm.com  Sat Jun  8 05:52:09 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Sat, 08 Jun 2013 13:52:09 +0800
Subject: SLA feature for storage I/O bandwidth
In-Reply-To: <666894327.19206560.1370442415368.JavaMail.root@redhat.com>
References: <51AC6FDE.5030007@linux.vnet.ibm.com>
	<51AEF2B6.6070503@linux.vnet.ibm.com>
	<26148712.4071809.1370423021002.JavaMail.root@redhat.com>
	<51AF0F87.2000809@linux.vnet.ibm.com>
	<666894327.19206560.1370442415368.JavaMail.root@redhat.com>
Message-ID: <51B2C689.7060801@linux.vnet.ibm.com>

Hi Doron,

Thanks for your suggestion. I add the vDisk profile in my design and 
define what parameters related to mom auto tuning  should be included in 
it. I also make the auto tuning policy simpler than before and match the 
profile design. The tuning will happen in each host by MOM. MOM relies 
on the collected IO statistics of vDisk  whose related VM residing on 
this host.

When vDisk is exported and imported to another SD, I think the profile 
related to vDisk should be reselected. This is because that the backend 
storage has changed and the profile should be selected from the profiles 
in new SD.

Could you take a look? You may also give comments on auto tuning policy. 
Thanks.

Best regards,
Mei Liu



On 06/05/2013 10:26 PM, Doron Fediuck wrote:
> Hi Mei Liu,
> sorry for the delay.
>
> I agree that the advanced approach is getting complicated, and
> that we can allow mom to control parts of it.
> Having that said, I'd still want you to look at the bigger picture
> of a VM QoS.
> Please refer to the discussion we had on network QoS, where we
> managed to define it in a way which going forward will be more
> user friendly on the one hand, but can be translated into mom-policy
> on the other hand.
> See: http://lists.ovirt.org/pipermail/engine-devel/2013-June/004764.html
>
> As you can see there, we defined a network profile, which includes a QoS
> element. Going forward this element will be taken from a policy, which
> will hold additional QoS aspects such as I/O, CPU and memory. In addition
> to that, the policy may include a reference to a quota allocation as well
> as other SLA related properties such as HA.
>
> So in this light, I'd like to see a disk-profile, which holds the QoS element
> and possibly other storage related properties, such as mom-auto-tune arguments.
> So once an admin sets this profile, the VM creator (or disk creator) can
> choose the relevant profile he needs for the disk and assign it to the disk.
> Once we have this in place, the engine will provide this policy to mom on
> VM creation, and mom will enforce I/O (in this case, but also network and
> others going forward) on the VM based on the policy it got. If the policy
> allows I/O auto-tuning, then mom will manage it for that disk.
>
> The benefits to this approach are that it's more holistic from system
> perspective. It should allow easy management and simple defaults which
> users should not deal with unless they have the relevant permissions such
> as disk creators.
>
> So in this context, I'd love to see a definition of disk or image profile,
> and the storage QoS element it holds. The VDSM API should already support
> policy updates for mom, so we only need to introduce the libvirt API elements.
>
> Feedback is more than welcomed.
> Doron
>
> ----- Original Message -----
>> From: "Mei Liu" <liumbj at linux.vnet.ibm.com>
>> To: "Allon Mureinik" <amureini at redhat.com>
>> Cc: "Doron Fediuck" <dfediuck at redhat.com>, arch at ovirt.org
>> Sent: Wednesday, June 5, 2013 1:14:31 PM
>> Subject: Re: SLA feature for storage I/O bandwidth
>>
>> On 06/05/2013 05:03 PM, Allon Mureinik wrote:
>>> Hi Mei,
>>>
>>> How are you treating shared disks?
>>> Is the limitation defined per disk (as a total), or per disk-vm relation?
>>>
>> The existing API in libvirt is to limit the  per vm's vDisk. e.g. hda of VM1
>>
>> So the limitation is on per vm's vDisk, if we don't consider quota for
>> SD in the design (the basic design). That limitation value is adjusted
>> based on if backend storage(localfs, nfs, glusterfs...) is heavily used.
>>
>> SD's IO bandwidth quota is to constrain the sum of vDisk limitation's
>> lower bound(min value of limitation, it is similar to the concept of
>> reserving). These vDisks use the volumes in the SD.
>>
>>
>>> ------------------------------------------------------------------------
>>>
>>>      *From: *"Mei Liu" <liumbj at linux.vnet.ibm.com>
>>>      *To: *"Doron Fediuck" <dfediuck at redhat.com>
>>>      *Cc: *arch at ovirt.org
>>>      *Sent: *Wednesday, June 5, 2013 11:11:34 AM
>>>      *Subject: *Re: SLA feature for storage I/O bandwidth
>>>
>>>      Hi Doron,
>>>      After the discussion, we found that the last version of design is
>>>      a little complex, so I simplified it and post it to
>>>
>>>      http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth
>>>
>>>      We want to let MOM in each host decides how to tune the bandwidth
>>>      limit of vms in that host instead of letting engine to make the
>>>      whole decision based on statistics from vms in diverse hosts.
>>>      Maybe we can consider starting from the basic one in wiki.
>>>
>>>      Thanks & best regards,
>>>      Mei Liu(Rose)
>>>
>>>      -------- Original Message --------
>>>      Subject: 	Re: SLA feature for storage I/O bandwidth
>>>      Date: 	Mon, 03 Jun 2013 18:28:46 +0800
>>>      From: 	Mei Liu <liumbj at linux.vnet.ibm.com>
>>>      To: 	Doron Fediuck <dfediuck at redhat.com>
>>>      CC: 	arch at ovirt.org
>>>
>>>
>>>
>>>      On 05/29/2013 11:34 PM, Doron Fediuck wrote:
>>>      > ----- Original Message -----
>>>      >> From: "Mei Liu"<liumbj at linux.vnet.ibm.com>
>>>      >> To: "Dave Neary"<dneary at redhat.com>
>>>      >> Cc:arch at ovirt.org
>>>      >> Sent: Wednesday, May 29, 2013 11:35:12 AM
>>>      >> Subject: Re: SLA feature for storage I/O bandwidth
>>>      >>
>>>      >> On 05/29/2013 03:42 PM, Dave Neary wrote:
>>>      >>> Hi Mei Liu,
>>>      >>>
>>>      >>> On 05/28/2013 10:18 AM, Mei Liu wrote:
>>>      >>>> I created a drafted wiki page on  design of storage I/O bandwidth
>>>      >>>> SLA in
>>>      >>>> the following link:
>>>      >>>>
>>>      >>>>http://www.ovirt.org/SLA_for_storage_resource  .
>>>      >>>>
>>>      >>>> I will appreciate the efforts if anyone who works on ovirt engine,
>>>      >>>> vdsm
>>>      >>>> or mom could give some comments. TIA.
>>>      >>> Just out of interest - which version of oVirt are you targeting
>>>      >>> this
>>>      >>> for? Can I assume that it's for post-3.3? Today is officially 3.3.
>>>      >>> feature freeze (but we have a release meeting later to discuss
>>>      >>> that).
>>>      >>>
>>>      >>> Thanks,
>>>      >>> Dave.
>>>      >>>
>>>      >> Hi Dave,
>>>      >> The basic i/o tune functionality for vdsm is almost ready. However,
>>>      >> there is nothing written on the Engine side and no policy for
>>>      >> automatic
>>>      >> tuning is applied yet.
>>>      >> I am not sure if the basic functionality can target 3.3.
>>>      >>
>>>      >>
>>>      >> Best regards,
>>>      >> Mei Liu
>>>      >>
>>>      > Hi Mey Liu,
>>>      > I'm still going over the wiki, but a few things we need to consider;
>>>      > First of all QoS for storage I/O bandwidth is a part of a larger SLA
>>>      > policy which may include network QoS, CPU and memory QoS and the
>>>      > quota
>>>      > we implement today.
>>>      >
>>>      > So first of all we need to make sure your design does not conflict
>>>      > the other QoS parts, which is what I'm looking into now.
>>>      >
>>>      > Additionally, using the quota term is confusing as oVirt already has
>>>      > quota today, and cpu API has his own quota definition. So please try
>>>      > to come up with a different terminology.
>>>      >
>>>      > I like your idea of setting an initial value but I need some more
>>>      > time
>>>      > for it to come up with my insights.
>>>      > Also, I completely agree with your concept of letting mom handle
>>>      > it in host level. We need to verify it does not break anything
>>>      > related
>>>      > to SPM. This is something we need to synchronize with the storage
>>>      > guys.
>>>      >
>>>      > Looking into the engine area, we should start thinking on how this
>>>      > will
>>>      > be supported in the main storage entities and VM / template /
>>>      > instance
>>>      > entities. So you may want to add a section on this as well. This
>>>      > leads
>>>      > me to think of importing and exporting a VM which may want to
>>>      > maintain
>>>      > the defined IO QoS. Any thoughts around it?
>>>      >
>>>      > Doron
>>>      >
>>>      Hi Doron,
>>>      Thanks for your questions and insightful thoughts. They are really
>>>      inspiring.
>>>
>>>      I updated the design in
>>>      http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth  .
>>>      This time, I add  storage I/O bandwidth control according to the quota
>>>      design and the SLA tries to ensure the reserved bandwidth for vDisks.
>>>      It requires the engine  to make the tuning decision, since vm uses the
>>>      SD volumes may  reside on different hosts and only engine can obtain
>>>      the
>>>      global information.
>>>      I think this design will not lead to problem when importing or
>>>      exporting
>>>      a VM.
>>>
>>>      I will appreciate if you can take a look at the new design.
>>>
>>>      Best regards,
>>>      Mei Liu (Rose)
>>>
>>>      _______________________________________________
>>>      Arch mailing list
>>>      Arch at ovirt.org
>>>      http://lists.ovirt.org/mailman/listinfo/arch
>>>
>>>
>>>
>>>
>>>
>>>      _______________________________________________
>>>      Arch mailing list
>>>      Arch at ovirt.org
>>>      http://lists.ovirt.org/mailman/listinfo/arch
>>>
>>>
>>



From mpastern at redhat.com  Sun Jun  9 07:48:07 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 09 Jun 2013 10:48:07 +0300
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <350995831.12892634.1370560204679.JavaMail.root@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
Message-ID: <51B43337.5000104@redhat.com>


Hi Ayal,

On 06/07/2013 02:10 AM, Ayal Baron wrote:
> 
> 
> ----- Original Message -----
>> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
>>> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
>>>> Hi Deepak,
>>>>
>>>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
>>>>> Hi All,
>>>>>     Per the previous mail from Ayal.. I am refering to the flow below...
>>>>>
>>>>> The backup flow should be (all api calls are engine REST API):
>>>> I disagree with this, by following this pattern you move internal
>>>> logic (even more than this given implementation specific logic) to
>>>> the api users instead of abstracting it and creating solid api for
>>>> backups,
> 
> This is not internal logic.
> I think what is not clear here is that we're *not* modelling a backup flow.
> We're modelling a flow to allow a user to expose an existing disk to a virt-app / physical appliance, while the VM which owns the disk is still running.

in this case this exposure is relevant for the backup and is a part of the
disk backup process.

> There can be a horde of uses for this, off the top of my head:
> 1. obviously backup
> 2. scan disk for viruses externally
> 3. scan disk for inventory management
> There will be many more which users come up with.

so?, i can't see why this is relevant? you can reuse this code in the backend later
when you need it for the new functionalities, - this is basic coding practice, and
we do that across the backend by reusing internal commands.

i can give you at least one example where bad design of the backend command
forces backend clients such as api, UI, etc. to implement same logic several
times manually without having any ability to perform atomic operation against
a backend:

snapshot.restore()
==================

you have to perform two separate steps in api against a backend:

1. VdcActionType.TryBackToAllSnapshotsOfVm
2. VdcActionType.RestoreAllSnapshots

well, we all developers and can do it once so api/UI/etc. users will see this as a
single operation and save the pain of learning restore() logic and running two
separate commands for restore to happen.

in your case this much worse, case you expect every api user to *learn vendor
specific* logic for backup/foo() and implement the steps + understand the responses
of each step and act accordingly (and i'm not even mentioning the atomic ops. here)

i apologise, but this is a bad thing to do, this is not how public api should look like.

> 
> Designing the API around the specific usage (policy) is limiting and will lead to a lot more API calls due to the need to add new ones for every new usage someone thinks of. 

that's what you suggested to begin with (and i'm opposed to it) -
e.g leaving "exposer of existing disk to a virt-app / physical appliance"
in the api

> 
> The API should provide users with useful mechanisms.
> In this case what we need is the ability to get a point in time of a disk available for read (and in some cases write) on one of the hosts (and later on possibly over the net as well, assuming we secure it).
> I do not always want to take a snapshot for example, I want to be able to use the API to expose an existing snapshot.
> Once we add live merge, user may *choose* to merge immediately after backup is finished or to keep the snapshot (policy).

these all are implementation details that can be easily hidden by adding
extra parameters to parameters holder as i described bellow,

actually this is exactly where abstraction shines and only proves my point.

> 
> 
> 
> 
>>>>
>>>> +++++++++ several api calls for single backup ++++++++++++++
>>>>
>>>>
>>>> cons:
>>>> ====
>>>>
>>>> 1. user have to be aware of internal (implementation specific) flow of
>>>> each provider in order to be able creating backup
>>>>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
>>>> 2. no flexibility for backup provider to change the backup flow (as it
>>>> would be considered an api break)
>>>> 3. no atomic resolution for the /backup failures
>>>>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
>>>>         that user have to del(a) manually and if it's deeper in the flow
>>>>         every api user will have to implement own rollbacks to the steps
>>>>         that took
>>>>         place before the failure.
>>>> 4. backward compatibility becomes an issue when exposing internals in such
>>>> a low level.
>>>> 5. new features expose becomes much complex task cause they may require
>>>> performing extra
>>>>    step/s in a middle.
>>>>
>>>> pros:
>>>> ====
>>>>
>>>> can't think of any ..., cause it doesn't give you flexibility, but creates
>>>> unnatural
>>>> (to public APIs) complexity.
>>>
>>> I was able to dig the below from one of the older mails. I had exactly this
>>> Q on why we need separate APIs Vs 1 and this is what Ayal had to say
>>>
>>> Deepak asked:
>>>
>>> /Wouldn't it be better if there was just 1 REST API for startign backup
>>> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
>>> internally caused the above individual APIs to get called ?
>>> Why do we have to expose each of the steps in backup process as
>>> individual REST APIs ?
>>> /
>>>
>>> Ayal said:
>>>
>>> /1. Because a single API assumes that I always want to backup everything
>>> which isn't necessarily true (normally backup policy for system disk and
>>> data disks are different)
>>
>> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
>>
>>> 2. Going forward we will have disk level snapshot in which case the backup
>>> API would have to change.
>>
>> same
>>
>>> 3. You could always return the vm config per disk when calling "prepare
>>> backup disk" which would be a bit redundant now but once we have disk
>>> level snapshots it would be more relevant and this way remove 1 API call
>>> now.
>>> Later on when we do have disk level snaps it could be an option to the
>>> command to take the snap or something I guess.
>>
>> it's only proves my point, we can/should hide this under api abstraction
>> rather than forcing users using different flows now and then.
>>
>>> /
>>> So i kind of agreed to what Ayal said... from the perspective that it
>>> provides flexibility to the end user using the API, as to how he/she wants
>>> to script/use it.
>>> What do you think ?  I just wanted to put the above here... so that we are
>>> sure we can considering all aspects before making the decision on 1 Vs
>>> many APIs.
>>
>> i've mentioned this before, - i don't think this is about flexibility, if you
>> want to allow
>> user doing several things differently, - expose several api for same thing
>> (like method overloads),
>> but it doesn't mean that we should be exposing implementation internals to
>> end user.
>>
>>>
>>> 1 API provides simplicity, but getting flexibility using 1 API would mean
>>> passing addnl params.. for eg> If i want to backup a particular vmdisk
>>> only, then the xml below would change, rite ?
>>
>> no, you misunderstood, see url [1] from my previous eamil, you already doing
>> this for specific disk - yyy
>>
>> [1] /api/vms/xxx/disks/yyy/backups
>>
>>> But looks like either way, its possible to get things done...
>>> I personally don't have much experience in the area of REST APIs to take
>>> the call on whether we need 1 or multiple APIs... so I am still
>>> inconclusive here (sorry about that!)
>>> All i can say is that with 1 API approach too, we can get flexibility.
>>>
>>>
>>>>
>>>>
>>>> +++++++++ single api call for single backup ++++++++++++++
>>>>
>>>> cons:
>>>> ====
>>>>
>>>> nothing comes to my mind, are there any?
>>>>
>>>> pros:
>>>> ====
>>>>
>>>> 1. user is not aware of any backup internals and can consume different
>>>> /backup
>>>>    providers using very same api and supplying parameters relevant for the
>>>>    provider
>>>>    he'd like to use.
>>>> 2. backup providers can change/extend internal flow as much as they
>>>> want/need, only
>>>>    by modifying internal engine driver, user using abstraction over the
>>>>    backup api
>>>>    won't feel the difference.
>>>> 3. backup providers can execute backup as single atomic operation and take
>>>> care
>>>>    for rollback cleanups in the transaction.
>>>> 4. backup providers can easily maintain backward compatibility over
>>>> abstraction
>>>> 5. adding new features that require flow extension can be easily hidden
>>>> under
>>>>    the abstraction layer.
>>>>
>>>> also this is a common practice to expose abstraction layer to public,
>>>> while
>>>> vendors only have to implement the driver in order to be supported.
>>>>
>>>>
>>>>> 1. create vm snapshot (the api call exists today)
>>>>> 2. get VM Config (new API)
>>>>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
>>>>> return: paths to device on host as well as map of changed blocks) -
>>>>> this should be called for every disk that needs to be backed up. Note
>>>>> that VM snapshot takes a snap of *all* disks of the VM
>>>>> 4. attach disk to backup vm (the api call exists today.  This assumes
>>>>> virt app) - also has to be called per disk to back up
>>>>> 5. At this point backup app would do the backup
>>>>>
>>>> this can be easily done under the single api call:
>>>> =================================================
>>>>
>>>> POST /api/vms/xxx/disks/yyy/backups
>>>>
>>>> <backup>
>>>>   <host id=aaa> #3
>>>>   <backup_vm id=bbb> #1, #2
>>>>   <delete_snapshot>true|false<delete_snapshot/> #7
>>>> <backup/>
>>>>
>>>> 1. create vm snapshot => you already in context of vm, trigger snapshot on
>>>> it
>>>> 2. get VM Config => you already in context of vm, collect relevant meta
>>>> 3. prepare backup disk => you in context of the disk already, do whatever
>>>> is needed
>>>> 4. attach disk to backup vm => you have backup_vm id in the request body
>>>> 5. do the actual backup
>>>>
>>>> does this makes sense?
>>>>
>>>>> 5) detach disk (symmetric to 4)
>>>>> 6. teardown backup disk (symmetric to 3)
>>>>> 7. delete snap - This can only be called if VM is down today and is
>>>>> not mandatory in any event.  Once we have live merge, it should be
>>>>> policy driven (user should be able to choose whether to keep or delete)
>>>>>
>>>>> Questions...
>>>>>
>>>>> 1) My current implmn of prepareBackupDisk in VDSM (pls see
>>>>> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
>>>>> returns drive and cbtinfo as dicts back to engine. The understnading here
>>>>> is that the drive returned will be a dict which has all the necessary
>>>>> info
>>>>> to represent a NBD drive to libvirt... hence this drive will be passed
>>>>> as-is in the "attach disk to VM" REST API. Is this correct ?
>>>>>
>>>>> Note that i cannot return a disk UUID (like VDSM does for create volume
>>>>> case).. because in preparebackupdisk case the image is exported usign
>>>>> qemu-nbd
>>>>> as a block device over the network... and hence its not a regular
>>>>> vdsm-type disk, hence no UUID, Agree ?
>>>>>
>>>>> 2) In "attach disk to VM" REST API user will pass the drive dict which
>>>>> he/she got as part of preparebackupdisk api.. as-is
>>>>> and VDSM will need to add support for accepting NBD disk as a valid
>>>>> disktype in VM.hotplugDisk() code of VDSM.
>>>
>>>>>
>>>>> The ability to add a network disk is present in libvirtvm.py... as part
>>>>> of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
>>>>> but it doesn't work at the API.py level.. its only from prepareImage
>>>>> onwards... hence the need to modify API.py to accept NBD disk type and
>>>>> connect
>>>>> to the libvirtvm.py appropriately
>>>>>
>>>>> Is this acceptable.. otherwise there is no good way to pass the NBD
>>>>> drive's info back to VDSM as part of existing "attach disk to VM" API
>>>>> Also does the existing "attach disk to VM" API work if a pre-loaded drive
>>>>> dict/hashmap is provided by the user. This will have drive->type =
>>>>> network
>>>>> and not file/blcok as is currently supported
>>>
>>> Mike,
>>>     Could you provide your views on this pls ? How do we represent the NBD
>>>     disk and pass it back to VDSM as part of 'hotplugDisk' API
>>> Currently hotplugDisk doesn't support drive of type = network!
>>>
>>>>>
>>>>> 3) After "attach disk to backup VM" REST API step.. the understnading is
>>>>> that some backup vendor specific API will be called to tell the
>>>>> backup appln....
>>>>>     -- Block device (attached to the VM) to be used as src for backup
>>>>>     -- CBT info of this blcok device (which was recd. as part of
>>>>>     prepareBackupDisk API)
>>>>> Is this correct ?
>>>>>
>>>>> thanx,
>>>>> deepak
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>>


-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From iheim at redhat.com  Sun Jun  9 10:28:06 2013
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 09 Jun 2013 13:28:06 +0300
Subject: gerrit will be upgraded now
Message-ID: <51B458B6.8070306@redhat.com>



From iheim at redhat.com  Sun Jun  9 10:31:07 2013
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 09 Jun 2013 13:31:07 +0300
Subject: gerrit will be upgraded now
In-Reply-To: <51B458B6.8070306@redhat.com>
References: <51B458B6.8070306@redhat.com>
Message-ID: <51B4596B.7060008@redhat.com>

On 06/09/2013 01:28 PM, Itamar Heim wrote:
>
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch

gerrit upgrade complete, please report any issues


From amureini at redhat.com  Sun Jun  9 11:45:37 2013
From: amureini at redhat.com (Allon Mureinik)
Date: Sun, 9 Jun 2013 07:45:37 -0400 (EDT)
Subject: gerrit will be upgraded now
In-Reply-To: <51B4596B.7060008@redhat.com>
References: <51B458B6.8070306@redhat.com> <51B4596B.7060008@redhat.com>
Message-ID: <409555242.458868.1370778337336.JavaMail.root@redhat.com>

The new version and the old version have an annoying difference.
In the older version, in the main screen, verified (V) was the second-right column, and reviewed (R) the right most.
In the newer version, verified (V) is right column, and code reviewed (CR) the second right.

Is this configurable?
If so, it's would be convenient to get it back.
If not, all - FYI.
Don't do the mistake I did and try to merge a patch that's merely verified :-)

----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "infra" <infra at ovirt.org>, "arch" <arch at ovirt.org>
> Sent: Sunday, June 9, 2013 1:31:07 PM
> Subject: Re: gerrit will be upgraded now
> 
> On 06/09/2013 01:28 PM, Itamar Heim wrote:
> >
> > _______________________________________________
> > Arch mailing list
> > Arch at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/arch
> 
> gerrit upgrade complete, please report any issues
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 


From abaron at redhat.com  Mon Jun 10 22:38:43 2013
From: abaron at redhat.com (Ayal Baron)
Date: Mon, 10 Jun 2013 18:38:43 -0400 (EDT)
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51B43337.5000104@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
	<51B43337.5000104@redhat.com>
Message-ID: <1433156354.14066468.1370903923309.JavaMail.root@redhat.com>



----- Original Message -----
> 
> Hi Ayal,
> 
> On 06/07/2013 02:10 AM, Ayal Baron wrote:
> > 
> > 
> > ----- Original Message -----
> >> On 06/05/2013 02:14 PM, Deepak C Shetty wrote:
> >>> On 06/05/2013 03:58 PM, Michael Pasternak wrote:
> >>>> Hi Deepak,
> >>>>
> >>>> On 05/30/2013 06:07 PM, Deepak C Shetty wrote:
> >>>>> Hi All,
> >>>>>     Per the previous mail from Ayal.. I am refering to the flow
> >>>>>     below...
> >>>>>
> >>>>> The backup flow should be (all api calls are engine REST API):
> >>>> I disagree with this, by following this pattern you move internal
> >>>> logic (even more than this given implementation specific logic) to
> >>>> the api users instead of abstracting it and creating solid api for
> >>>> backups,
> > 
> > This is not internal logic.
> > I think what is not clear here is that we're *not* modelling a backup flow.
> > We're modelling a flow to allow a user to expose an existing disk to a
> > virt-app / physical appliance, while the VM which owns the disk is still
> > running.
> 
> in this case this exposure is relevant for the backup and is a part of the
> disk backup process.

That is an *example* of a specific usage. You should not program to a specific usage, rather build mechanisms that can be reused.

> 
> > There can be a horde of uses for this, off the top of my head:
> > 1. obviously backup
> > 2. scan disk for viruses externally
> > 3. scan disk for inventory management
> > There will be many more which users come up with.
> 
> so?, i can't see why this is relevant? you can reuse this code in the backend
> later
> when you need it for the new functionalities, - this is basic coding
> practice, and
> we do that across the backend by reusing internal commands.

The point is we're adding these functionalities *right now* without having to duplicate or write usage specific APIs.

> 
> i can give you at least one example where bad design of the backend command
> forces backend clients such as api, UI, etc. to implement same logic several
> times manually without having any ability to perform atomic operation against
> a backend:
> 
> snapshot.restore()
> ==================
> 
> you have to perform two separate steps in api against a backend:
> 
> 1. VdcActionType.TryBackToAllSnapshotsOfVm
> 2. VdcActionType.RestoreAllSnapshots

The fact that there are bad APIs doesn't reflect on our current proposal.
In the above example I don't even understand the names of the API calls, let alone what they do and why the separation was made.  It may very well be that the separation here is wrong, but that does not reflect anything on the API we're talking about here.

> 
> well, we all developers and can do it once so api/UI/etc. users will see this
> as a
> single operation and save the pain of learning restore() logic and running
> two
> separate commands for restore to happen.
> 
> in your case this much worse, case you expect every api user to *learn vendor
> specific* logic for backup/foo() and implement the steps + understand the
> responses
> of each step and act accordingly (and i'm not even mentioning the atomic ops.
> here)

I think there is some misunderstanding here.
The 'users' *are* the backup vendors.
These APIs are intended directly for backup vendors who will integrate once with the API.
Users wanting to backup or restore will go to the vendors application not to oVirt.

> 
> i apologise, but this is a bad thing to do, this is not how public api should
> look like.
> 
> > 
> > Designing the API around the specific usage (policy) is limiting and will
> > lead to a lot more API calls due to the need to add new ones for every new
> > usage someone thinks of.
> 
> that's what you suggested to begin with (and i'm opposed to it) -
> e.g leaving "exposer of existing disk to a virt-app / physical appliance"
> in the api

I don't understand this comment. What are you opposed to and why?

> 
> > 
> > The API should provide users with useful mechanisms.
> > In this case what we need is the ability to get a point in time of a disk
> > available for read (and in some cases write) on one of the hosts (and
> > later on possibly over the net as well, assuming we secure it).
> > I do not always want to take a snapshot for example, I want to be able to
> > use the API to expose an existing snapshot.
> > Once we add live merge, user may *choose* to merge immediately after backup
> > is finished or to keep the snapshot (policy).
> 
> these all are implementation details that can be easily hidden by adding
> extra parameters to parameters holder as i described bellow,
> 
> actually this is exactly where abstraction shines and only proves my point.

Parameters that do what exactly?
Let's describe this more clearly. In a single API this would be (pseudocode):

prepareDiskForBackupOrRestore(int hostID, int diskID, boolean autoAttach, int vmID, boolean createSnapshot, int snapshotID, boolean autoMerge, boolean writeable)

tearDownDisk(...)

hostID         - which host to expose the disk on
diskID         - which disk should be exposed
autoAttach     - should disk be attached to a VM or not
vmID           - if VM should be attached then which VM to attach it to
createSnapshot - should operation first create a snapshot or not
snapshotID     - if createSnapshot is false then which snapshot should be used
autoMerge      - should snapshot be automatically merged when operation ends (may only be 'True' if createSnapshot is true) - this would fail today if the VM we're backing up is running since we do not yet support live merge.
writeable      - should disk be exposed read only or read write (may only be 'True' if autoMerge is False)

Just by looking at the API you can see that it's wrong.  All these variables change the entire flow of the method and should clearly be broken down into separate calls.
Note also that any assumption of atomicity here is false since you cannot guarantee atomicity of all of these operations (e.g. once snapshot creation completes and operation moves to the next phase you cannot 'rollback' as the disks are no longer in the same state).

The alternative is:

getSnapConfig  (int snapshotID) *
mountDisk      (int hostId, int diskId, int snapshotId, boolean writeable)
unmountDisk    (...)

And for the rest of the options:
createSnapshot (already exists today and is optional in the process)
attachDisk     (already exists today and is optional in the process)
detachDisk     (already exists today and is optional in the process)
deleteSnapshot (already exists today and is optional in the process) - note that since this is separate, user can run this at a later point in time when the VM to backup is not running.

With the latter approach, there are 3 new API calls (compared with 2 in a monolithic approach) and we immediately support other usages such as:
scan disk for viruses externally
scan disk for inventory management
etc, without writing a single line of code nor bloating the API with every usage imaginable.



> 
> > 
> > 
> > 
> > 
> >>>>
> >>>> +++++++++ several api calls for single backup ++++++++++++++
> >>>>
> >>>>
> >>>> cons:
> >>>> ====
> >>>>
> >>>> 1. user have to be aware of internal (implementation specific) flow of
> >>>> each provider in order to be able creating backup
> >>>>    1.1 call add(a)->do(a)->del(a)->do(b)->etc.
> >>>> 2. no flexibility for backup provider to change the backup flow (as it
> >>>> would be considered an api break)
> >>>> 3. no atomic resolution for the /backup failures
> >>>>    3.1. failure at do(b) of add(a)->do(a)->del(a)->do(b)->etc. will mean
> >>>>         that user have to del(a) manually and if it's deeper in the flow
> >>>>         every api user will have to implement own rollbacks to the steps
> >>>>         that took
> >>>>         place before the failure.
> >>>> 4. backward compatibility becomes an issue when exposing internals in
> >>>> such
> >>>> a low level.
> >>>> 5. new features expose becomes much complex task cause they may require
> >>>> performing extra
> >>>>    step/s in a middle.
> >>>>
> >>>> pros:
> >>>> ====
> >>>>
> >>>> can't think of any ..., cause it doesn't give you flexibility, but
> >>>> creates
> >>>> unnatural
> >>>> (to public APIs) complexity.
> >>>
> >>> I was able to dig the below from one of the older mails. I had exactly
> >>> this
> >>> Q on why we need separate APIs Vs 1 and this is what Ayal had to say
> >>>
> >>> Deepak asked:
> >>>
> >>> /Wouldn't it be better if there was just 1 REST API for startign backup
> >>> which would take all the necessary inputs (VM uuid, hostid, disk(s)) and
> >>> internally caused the above individual APIs to get called ?
> >>> Why do we have to expose each of the steps in backup process as
> >>> individual REST APIs ?
> >>> /
> >>>
> >>> Ayal said:
> >>>
> >>> /1. Because a single API assumes that I always want to backup everything
> >>> which isn't necessarily true (normally backup policy for system disk and
> >>> data disks are different)
> >>
> >> modelling backup under /api/vms/xxx/disks/yyy/backups solves this.
> >>
> >>> 2. Going forward we will have disk level snapshot in which case the
> >>> backup
> >>> API would have to change.
> >>
> >> same
> >>
> >>> 3. You could always return the vm config per disk when calling "prepare
> >>> backup disk" which would be a bit redundant now but once we have disk
> >>> level snapshots it would be more relevant and this way remove 1 API call
> >>> now.
> >>> Later on when we do have disk level snaps it could be an option to the
> >>> command to take the snap or something I guess.
> >>
> >> it's only proves my point, we can/should hide this under api abstraction
> >> rather than forcing users using different flows now and then.
> >>
> >>> /
> >>> So i kind of agreed to what Ayal said... from the perspective that it
> >>> provides flexibility to the end user using the API, as to how he/she
> >>> wants
> >>> to script/use it.
> >>> What do you think ?  I just wanted to put the above here... so that we
> >>> are
> >>> sure we can considering all aspects before making the decision on 1 Vs
> >>> many APIs.
> >>
> >> i've mentioned this before, - i don't think this is about flexibility, if
> >> you
> >> want to allow
> >> user doing several things differently, - expose several api for same thing
> >> (like method overloads),
> >> but it doesn't mean that we should be exposing implementation internals to
> >> end user.
> >>
> >>>
> >>> 1 API provides simplicity, but getting flexibility using 1 API would mean
> >>> passing addnl params.. for eg> If i want to backup a particular vmdisk
> >>> only, then the xml below would change, rite ?
> >>
> >> no, you misunderstood, see url [1] from my previous eamil, you already
> >> doing
> >> this for specific disk - yyy
> >>
> >> [1] /api/vms/xxx/disks/yyy/backups
> >>
> >>> But looks like either way, its possible to get things done...
> >>> I personally don't have much experience in the area of REST APIs to take
> >>> the call on whether we need 1 or multiple APIs... so I am still
> >>> inconclusive here (sorry about that!)
> >>> All i can say is that with 1 API approach too, we can get flexibility.
> >>>
> >>>
> >>>>
> >>>>
> >>>> +++++++++ single api call for single backup ++++++++++++++
> >>>>
> >>>> cons:
> >>>> ====
> >>>>
> >>>> nothing comes to my mind, are there any?
> >>>>
> >>>> pros:
> >>>> ====
> >>>>
> >>>> 1. user is not aware of any backup internals and can consume different
> >>>> /backup
> >>>>    providers using very same api and supplying parameters relevant for
> >>>>    the
> >>>>    provider
> >>>>    he'd like to use.
> >>>> 2. backup providers can change/extend internal flow as much as they
> >>>> want/need, only
> >>>>    by modifying internal engine driver, user using abstraction over the
> >>>>    backup api
> >>>>    won't feel the difference.
> >>>> 3. backup providers can execute backup as single atomic operation and
> >>>> take
> >>>> care
> >>>>    for rollback cleanups in the transaction.
> >>>> 4. backup providers can easily maintain backward compatibility over
> >>>> abstraction
> >>>> 5. adding new features that require flow extension can be easily hidden
> >>>> under
> >>>>    the abstraction layer.
> >>>>
> >>>> also this is a common practice to expose abstraction layer to public,
> >>>> while
> >>>> vendors only have to implement the driver in order to be supported.
> >>>>
> >>>>
> >>>>> 1. create vm snapshot (the api call exists today)
> >>>>> 2. get VM Config (new API)
> >>>>> 3. prepare backup disk (new api, should probably accept: hostid, disk;
> >>>>> return: paths to device on host as well as map of changed blocks) -
> >>>>> this should be called for every disk that needs to be backed up. Note
> >>>>> that VM snapshot takes a snap of *all* disks of the VM
> >>>>> 4. attach disk to backup vm (the api call exists today.  This assumes
> >>>>> virt app) - also has to be called per disk to back up
> >>>>> 5. At this point backup app would do the backup
> >>>>>
> >>>> this can be easily done under the single api call:
> >>>> =================================================
> >>>>
> >>>> POST /api/vms/xxx/disks/yyy/backups
> >>>>
> >>>> <backup>
> >>>>   <host id=aaa> #3
> >>>>   <backup_vm id=bbb> #1, #2
> >>>>   <delete_snapshot>true|false<delete_snapshot/> #7
> >>>> <backup/>
> >>>>
> >>>> 1. create vm snapshot => you already in context of vm, trigger snapshot
> >>>> on
> >>>> it
> >>>> 2. get VM Config => you already in context of vm, collect relevant meta
> >>>> 3. prepare backup disk => you in context of the disk already, do
> >>>> whatever
> >>>> is needed
> >>>> 4. attach disk to backup vm => you have backup_vm id in the request body
> >>>> 5. do the actual backup
> >>>>
> >>>> does this makes sense?
> >>>>
> >>>>> 5) detach disk (symmetric to 4)
> >>>>> 6. teardown backup disk (symmetric to 3)
> >>>>> 7. delete snap - This can only be called if VM is down today and is
> >>>>> not mandatory in any event.  Once we have live merge, it should be
> >>>>> policy driven (user should be able to choose whether to keep or delete)
> >>>>>
> >>>>> Questions...
> >>>>>
> >>>>> 1) My current implmn of prepareBackupDisk in VDSM (pls see
> >>>>> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:backup-restore,n,z)
> >>>>> returns drive and cbtinfo as dicts back to engine. The understnading
> >>>>> here
> >>>>> is that the drive returned will be a dict which has all the necessary
> >>>>> info
> >>>>> to represent a NBD drive to libvirt... hence this drive will be passed
> >>>>> as-is in the "attach disk to VM" REST API. Is this correct ?
> >>>>>
> >>>>> Note that i cannot return a disk UUID (like VDSM does for create volume
> >>>>> case).. because in preparebackupdisk case the image is exported usign
> >>>>> qemu-nbd
> >>>>> as a block device over the network... and hence its not a regular
> >>>>> vdsm-type disk, hence no UUID, Agree ?
> >>>>>
> >>>>> 2) In "attach disk to VM" REST API user will pass the drive dict which
> >>>>> he/she got as part of preparebackupdisk api.. as-is
> >>>>> and VDSM will need to add support for accepting NBD disk as a valid
> >>>>> disktype in VM.hotplugDisk() code of VDSM.
> >>>
> >>>>>
> >>>>> The ability to add a network disk is present in libvirtvm.py... as part
> >>>>> of my GlusterSD work, sicne gluster is represented as a NBD to QEMU
> >>>>> but it doesn't work at the API.py level.. its only from prepareImage
> >>>>> onwards... hence the need to modify API.py to accept NBD disk type and
> >>>>> connect
> >>>>> to the libvirtvm.py appropriately
> >>>>>
> >>>>> Is this acceptable.. otherwise there is no good way to pass the NBD
> >>>>> drive's info back to VDSM as part of existing "attach disk to VM" API
> >>>>> Also does the existing "attach disk to VM" API work if a pre-loaded
> >>>>> drive
> >>>>> dict/hashmap is provided by the user. This will have drive->type =
> >>>>> network
> >>>>> and not file/blcok as is currently supported
> >>>
> >>> Mike,
> >>>     Could you provide your views on this pls ? How do we represent the
> >>>     NBD
> >>>     disk and pass it back to VDSM as part of 'hotplugDisk' API
> >>> Currently hotplugDisk doesn't support drive of type = network!
> >>>
> >>>>>
> >>>>> 3) After "attach disk to backup VM" REST API step.. the understnading
> >>>>> is
> >>>>> that some backup vendor specific API will be called to tell the
> >>>>> backup appln....
> >>>>>     -- Block device (attached to the VM) to be used as src for backup
> >>>>>     -- CBT info of this blcok device (which was recd. as part of
> >>>>>     prepareBackupDisk API)
> >>>>> Is this correct ?
> >>>>>
> >>>>> thanx,
> >>>>> deepak
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >>
> >> Michael Pasternak
> >> RedHat, ENG-Virtualization R&D
> >>
> 
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From mpastern at redhat.com  Sun Jun 16 10:29:11 2013
From: mpastern at redhat.com (Michael Pasternak)
Date: Sun, 16 Jun 2013 13:29:11 +0300
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <1433156354.14066468.1370903923309.JavaMail.root@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
	<51B43337.5000104@redhat.com>
	<1433156354.14066468.1370903923309.JavaMail.root@redhat.com>
Message-ID: <51BD9377.7070405@redhat.com>


Hi Ayal,

On 06/11/2013 01:38 AM, Ayal Baron wrote:
> I think there is some misunderstanding here.
> The 'users' *are* the backup vendors.
> These APIs are intended directly for backup vendors who will integrate once with the API.
> Users wanting to backup or restore will go to the vendors application not to oVirt.

Well this is whole different story, i could never guess that from the feature wiki,
obviously when speaking about public api, first thing that pops up is a ovirt user,
not backup vendor.

So AFAIU the backup process will be triggered/managed from the backup vendor app.,
in that case i'm perfectly fine with the current design.


cheers.

-- 

Michael Pasternak
RedHat, ENG-Virtualization R&D


From abaron at redhat.com  Mon Jun 17 12:27:20 2013
From: abaron at redhat.com (Ayal Baron)
Date: Mon, 17 Jun 2013 08:27:20 -0400 (EDT)
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51BD9377.7070405@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
	<51B43337.5000104@redhat.com>
	<1433156354.14066468.1370903923309.JavaMail.root@redhat.com>
	<51BD9377.7070405@redhat.com>
Message-ID: <1944585584.16867066.1371472040674.JavaMail.root@redhat.com>



----- Original Message -----
> 
> Hi Ayal,
> 
> On 06/11/2013 01:38 AM, Ayal Baron wrote:
> > I think there is some misunderstanding here.
> > The 'users' *are* the backup vendors.
> > These APIs are intended directly for backup vendors who will integrate once
> > with the API.
> > Users wanting to backup or restore will go to the vendors application not
> > to oVirt.
> 
> Well this is whole different story, i could never guess that from the feature
> wiki,
> obviously when speaking about public api, first thing that pops up is a ovirt
> user,
> not backup vendor.
> 
> So AFAIU the backup process will be triggered/managed from the backup vendor
> app.,
> in that case i'm perfectly fine with the current design.

Ack, so we'll fix the wiki to explain this better, sorry for the confusion, I had a feeling we're not talking about the same things here and I do agree that for an oVirt user rather than vendor it would be a pain in the $%@#.


> 
> 
> cheers.
> 
> --
> 
> Michael Pasternak
> RedHat, ENG-Virtualization R&D
> 


From liumbj at linux.vnet.ibm.com  Tue Jun 18 04:00:07 2013
From: liumbj at linux.vnet.ibm.com (Mei Liu)
Date: Tue, 18 Jun 2013 12:00:07 +0800
Subject: encounter problem to detect IO bandwidth congestion in ovirt storage
	domains
Message-ID: <51BFDB47.3040908@linux.vnet.ibm.com>

Hi,

I found that its a little difficult to detect IO bandwidth congestion in 
ovirt storage domains supported by NFS or GlusterFs.

For block based storage, it's easier to detect, since you can use some 
tool like iostat.For the case of file system based storage, it's much 
harder.

I investigate the existing solution. vsphere uses average IO latency to 
detect it. I propose a similar scheme in 
http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth . It 
simplifies the scheme by make the congestion decision on a single host 
instead of using the statistics from all the hosts use the backend 
storage. It doesn't need communication between hosts and maybe in phase 
two we can add communication and make a global decision.

  For now, it detects congestion viastatistics of vms using that backend 
storage in the local host(This info is collected through iostat in vm).  
It collects IO latency in such vms and compute an average latency for 
that backend storage. If it is higher than threshold, a congestion is found.

However, when I did testing for such a policy, I found that setting IO 
limit to a smaller value will make latency longer. That means if average 
latency exceeds that threshold and then our automatically tuning IO 
limit will be decreased which lead to average IO latency longer. Of 
course, this IO latency will exceed the threshold again and cause the IO 
limit be decreased. This will finally cause the IO limit to its lower 
bound.

This scheme has affected by the following reasons:
1. we collect statistic data from vms instead of host. (This is because 
it is hard to collect such info for remote storage like NFS, GlusterFS)
2.The IO limit affect the latency.
3. The threshold is a constant.
4 I also find that iostat's await(latency info) is not good enough since 
the latency is long for very light io or very heavy IO.


Does anybody get an idea or have experience on this? Suggestions are 
more than welcome. Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/arch/attachments/20130618/2961bc42/attachment.html>

From deepakcs at linux.vnet.ibm.com  Tue Jun 18 12:43:49 2013
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Tue, 18 Jun 2013 18:13:49 +0530
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <51BD9377.7070405@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
	<51B43337.5000104@redhat.com>
	<1433156354.14066468.1370903923309.JavaMail.root@redhat.com>
	<51BD9377.7070405@redhat.com>
Message-ID: <51C05605.1080404@linux.vnet.ibm.com>

Just an update :
    I posted new patchset using Paolo bonzini's qcow.py and shell script 
combo, that eventually helps create a new block device on the VDSM host. 
Would appreciate quick reviews. This is a intermediate step.. since we 
need to replace it with qemu-img once bonzini has added this support in 
there.

thanx,
deepak



From msivak at redhat.com  Tue Jun 18 16:30:51 2013
From: msivak at redhat.com (Martin Sivak)
Date: Tue, 18 Jun 2013 12:30:51 -0400 (EDT)
Subject: [MoM] Minimum guaranteed memory
In-Reply-To: <657854092.25416913.1371572298964.JavaMail.root@redhat.com>
Message-ID: <1325947852.25423586.1371573051023.JavaMail.root@redhat.com>

Hi,

I am working on a feature that will ensure that the VMs will have some configurable minimum amount of memory available regardless of the ballooning. The MoM changes I have so far are these two:

http://gerrit.ovirt.org/#/c/15800/ - This one teaches min and max functions to the MoM policy parser
http://gerrit.ovirt.org/#/c/15801/ - This one adds a new balloon_min property to accomplish the task

I also have a related patch proposed for vdsm (http://gerrit.ovirt.org/#/c/15799/).

The reason I am posting here is that I would like to talk about what should happen when MoM is running standalone without VDSM. The patch uses hardcoded 0 for the balloon_min in that case, but it might be nicer to read some value from libvirt's metadata for the controlled VM instead.

I haven't seen any places where we would be using it so I tried to invent my own. I would like to propose the following snippet:

<metadata>
    <ovirt>
        <guaranteedmemory>value_in_KiB</guaranteedmemory>
    </ovirt>
</metadata>

What do you think?

--
Martin Siv?k
msivak at redhat.com
Red Hat Czech
RHEV-M SLA / Brno, CZ



From msivak at redhat.com  Wed Jun 19 16:20:32 2013
From: msivak at redhat.com (Martin Sivak)
Date: Wed, 19 Jun 2013 12:20:32 -0400 (EDT)
Subject: Unifying units in memory setting and reporting
In-Reply-To: <1995278001.26299040.1371658366270.JavaMail.root@redhat.com>
Message-ID: <919166467.26306989.1371658832052.JavaMail.root@redhat.com>

Hi,

we have an issue with reporting data over xml-rpc. It supports only 32bit integers and we ran into the limitation in https://bugzilla.redhat.com/show_bug.cgi?id=974917

It is probably caused by us using KiB units when reporting balloon memory (the limit is 2047 GiB which is high but reachable).

According to my understanding of the sources we already use MiB when creating the VM and so I propose we unify that and use megabytes in reporting as well. This will probably require change to all parts: VDSM, MoM and engine.

I have created a bug to track it: https://bugzilla.redhat.com/show_bug.cgi?id=975945

Regarding implementation, please do not be fooled by the variable names balloon_max/balloon_cur actually hold the maximum and current amount of memory available to the VM. We (me and Doron) were confused by it this morning while investigating for another SLA feature regarding minimal guaranteed memory.

Regards

--
Martin Siv?k
msivak at redhat.com
Red Hat Czech
RHEV-M SLA / Brno, CZ



From alonbl at redhat.com  Sun Jun 23 07:35:17 2013
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Jun 2013 03:35:17 -0400 (EDT)
Subject: Feedback image at ovirt-engine
In-Reply-To: <1451490611.11604758.1371972084041.JavaMail.root@redhat.com>
Message-ID: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>

Hello,

There was a request to add feedback image to the webadmin portal, this has been implemented[1] using the new branding plugins.

My question is which URL to redirect user when pressing the feedback image, currently I just put http://www.ovirt.org.

Using mailto: to a mailing list is something I do not like as:
1. Most people do not have mailto: handler anymore.
2. User will not know that he is forwarded to a mailing list and will wait for a response.
3. There is no way to enforce format (subject, titles).

But any URL is possible... including empty to disable the feature.

What do you think?
Alon

[1] http://gerrit.ovirt.org/#/c/15984/


From iheim at redhat.com  Sun Jun 23 15:21:43 2013
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Jun 2013 18:21:43 +0300
Subject: Feedback image at ovirt-engine
In-Reply-To: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>
References: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>
Message-ID: <51C71287.4030705@redhat.com>

On 06/23/2013 10:35 AM, Alon Bar-Lev wrote:
> Hello,
>
> There was a request to add feedback image to the webadmin portal, this has been implemented[1] using the new branding plugins.
>
> My question is which URL to redirect user when pressing the feedback image, currently I just put http://www.ovirt.org.
>
> Using mailto: to a mailing list is something I do not like as:
> 1. Most people do not have mailto: handler anymore.
> 2. User will not know that he is forwarded to a mailing list and will wait for a response.
> 3. There is no way to enforce format (subject, titles).
>
> But any URL is possible... including empty to disable the feature.
>
> What do you think?
> Alon
>
> [1] http://gerrit.ovirt.org/#/c/15984/
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
>

I do like mailto, even if the mail handler is disabled, its easy to 
understand what should i do to get more help, send feedback, etc.


From alonbl at redhat.com  Sun Jun 23 15:22:22 2013
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Jun 2013 11:22:22 -0400 (EDT)
Subject: Feedback image at ovirt-engine
In-Reply-To: <51C71287.4030705@redhat.com>
References: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>
	<51C71287.4030705@redhat.com>
Message-ID: <84207001.11623536.1372000942916.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "arch" <arch at ovirt.org>
> Sent: Sunday, June 23, 2013 6:21:43 PM
> Subject: Re: Feedback image at ovirt-engine
> 
> On 06/23/2013 10:35 AM, Alon Bar-Lev wrote:
> > Hello,
> >
> > There was a request to add feedback image to the webadmin portal, this has
> > been implemented[1] using the new branding plugins.
> >
> > My question is which URL to redirect user when pressing the feedback image,
> > currently I just put http://www.ovirt.org.
> >
> > Using mailto: to a mailing list is something I do not like as:
> > 1. Most people do not have mailto: handler anymore.
> > 2. User will not know that he is forwarded to a mailing list and will wait
> > for a response.
> > 3. There is no way to enforce format (subject, titles).
> >
> > But any URL is possible... including empty to disable the feature.
> >
> > What do you think?
> > Alon
> >
> > [1] http://gerrit.ovirt.org/#/c/15984/
> > _______________________________________________
> > Arch mailing list
> > Arch at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/arch
> >
> 
> I do like mailto, even if the mail handler is disabled, its easy to
> understand what should i do to get more help, send feedback, etc.
> 

Please suggest what mailto.


From iheim at redhat.com  Sun Jun 23 15:25:15 2013
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Jun 2013 18:25:15 +0300
Subject: Feedback image at ovirt-engine
In-Reply-To: <84207001.11623536.1372000942916.JavaMail.root@redhat.com>
References: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>
	<51C71287.4030705@redhat.com>
	<84207001.11623536.1372000942916.JavaMail.root@redhat.com>
Message-ID: <51C7135B.1070103@redhat.com>

On 06/23/2013 06:22 PM, Alon Bar-Lev wrote:
>
>
> ----- Original Message -----
>> From: "Itamar Heim" <iheim at redhat.com>
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: "engine-devel" <engine-devel at ovirt.org>, "arch" <arch at ovirt.org>
>> Sent: Sunday, June 23, 2013 6:21:43 PM
>> Subject: Re: Feedback image at ovirt-engine
>>
>> On 06/23/2013 10:35 AM, Alon Bar-Lev wrote:
>>> Hello,
>>>
>>> There was a request to add feedback image to the webadmin portal, this has
>>> been implemented[1] using the new branding plugins.
>>>
>>> My question is which URL to redirect user when pressing the feedback image,
>>> currently I just put http://www.ovirt.org.
>>>
>>> Using mailto: to a mailing list is something I do not like as:
>>> 1. Most people do not have mailto: handler anymore.
>>> 2. User will not know that he is forwarded to a mailing list and will wait
>>> for a response.
>>> 3. There is no way to enforce format (subject, titles).
>>>
>>> But any URL is possible... including empty to disable the feature.
>>>
>>> What do you think?
>>> Alon
>>>
>>> [1] http://gerrit.ovirt.org/#/c/15984/
>>> _______________________________________________
>>> Arch mailing list
>>> Arch at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/arch
>>>
>>
>> I do like mailto, even if the mail handler is disabled, its easy to
>> understand what should i do to get more help, send feedback, etc.
>>
>
> Please suggest what mailto.
>

mailto: users at ovirt.org
not sure i would even put a default subject


From alonbl at redhat.com  Sun Jun 23 15:37:17 2013
From: alonbl at redhat.com (Alon Bar-Lev)
Date: Sun, 23 Jun 2013 11:37:17 -0400 (EDT)
Subject: Feedback image at ovirt-engine
In-Reply-To: <51C7135B.1070103@redhat.com>
References: <1721109520.11605189.1371972917704.JavaMail.root@redhat.com>
	<51C71287.4030705@redhat.com>
	<84207001.11623536.1372000942916.JavaMail.root@redhat.com>
	<51C7135B.1070103@redhat.com>
Message-ID: <1741491928.11624041.1372001837127.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "Alon Bar-Lev" <alonbl at redhat.com>
> Cc: "engine-devel" <engine-devel at ovirt.org>, "arch" <arch at ovirt.org>
> Sent: Sunday, June 23, 2013 6:25:15 PM
> Subject: Re: Feedback image at ovirt-engine
> 
> On 06/23/2013 06:22 PM, Alon Bar-Lev wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Itamar Heim" <iheim at redhat.com>
> >> To: "Alon Bar-Lev" <alonbl at redhat.com>
> >> Cc: "engine-devel" <engine-devel at ovirt.org>, "arch" <arch at ovirt.org>
> >> Sent: Sunday, June 23, 2013 6:21:43 PM
> >> Subject: Re: Feedback image at ovirt-engine
> >>
> >> On 06/23/2013 10:35 AM, Alon Bar-Lev wrote:
> >>> Hello,
> >>>
> >>> There was a request to add feedback image to the webadmin portal, this
> >>> has
> >>> been implemented[1] using the new branding plugins.
> >>>
> >>> My question is which URL to redirect user when pressing the feedback
> >>> image,
> >>> currently I just put http://www.ovirt.org.
> >>>
> >>> Using mailto: to a mailing list is something I do not like as:
> >>> 1. Most people do not have mailto: handler anymore.
> >>> 2. User will not know that he is forwarded to a mailing list and will
> >>> wait
> >>> for a response.
> >>> 3. There is no way to enforce format (subject, titles).
> >>>
> >>> But any URL is possible... including empty to disable the feature.
> >>>
> >>> What do you think?
> >>> Alon
> >>>
> >>> [1] http://gerrit.ovirt.org/#/c/15984/
> >>> _______________________________________________
> >>> Arch mailing list
> >>> Arch at ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/arch
> >>>
> >>
> >> I do like mailto, even if the mail handler is disabled, its easy to
> >> understand what should i do to get more help, send feedback, etc.
> >>
> >
> > Please suggest what mailto.
> >
> 
> mailto: users at ovirt.org
> not sure i would even put a default subject
> 

OK, done.


From iheim at redhat.com  Sun Jun 23 19:06:04 2013
From: iheim at redhat.com (Itamar Heim)
Date: Sun, 23 Jun 2013 22:06:04 +0300
Subject: Gerrit 2.5.4 upgrade
Message-ID: <51C7471C.70504@redhat.com>

just to sum up the highlights of upgrade from June 9th:

- Gerrit was upgraded to 2.5.4

- new REST API
http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.5.html#_rest_api

- Custom Dashboards
http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/user-custom-dashboards.html

- the inline email are now part of gerrit (we used to run a patched 2.4
   version with them)

- new prolog submit rules (still considering how can be best used)
http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/prolog-cookbook.html

- the mirroring plugin (to github.com/oVirt) is now a plugin (Thanks
   David)

- looking forward to gerrit 2.6 upgrade which promises "42x improvement
   on git clone and git fetch"

(I also remember it was supposed to improve copying approval flags on 
rebase, but couldn't find the reference).

Thanks,
    Itamar


From danken at redhat.com  Mon Jun 24 08:43:33 2013
From: danken at redhat.com (Dan Kenigsberg)
Date: Mon, 24 Jun 2013 11:43:33 +0300
Subject: Gerrit 2.5.4 upgrade
In-Reply-To: <51C7471C.70504@redhat.com>
References: <51C7471C.70504@redhat.com>
Message-ID: <20130624084333.GD22940@redhat.com>

On Sun, Jun 23, 2013 at 10:06:04PM +0300, Itamar Heim wrote:
> just to sum up the highlights of upgrade from June 9th:
> 
> - Gerrit was upgraded to 2.5.4
> 
> - new REST API
> http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.5.html#_rest_api
> 
> - Custom Dashboards
> http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/user-custom-dashboards.html
> 
> - the inline email are now part of gerrit (we used to run a patched 2.4
>   version with them)
> 
> - new prolog submit rules (still considering how can be best used)
> http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/prolog-cookbook.html
> 
> - the mirroring plugin (to github.com/oVirt) is now a plugin (Thanks
>   David)
> 
> - looking forward to gerrit 2.6 upgrade which promises "42x improvement
>   on git clone and git fetch"
> 
> (I also remember it was supposed to improve copying approval flags
> on rebase, but couldn't find the reference).

I like most of the improvements in this upgrade (particularly the quick
version selection in side-by-side view, dependencies remembered after
submit) but it's time to rant about

 Issue 1814:    Column-order "code-review/verified" different in list
 and detail-view

which would be fixed only in 2.8 :-(



From amureini at redhat.com  Mon Jun 24 08:58:00 2013
From: amureini at redhat.com (Allon Mureinik)
Date: Mon, 24 Jun 2013 04:58:00 -0400 (EDT)
Subject: Gerrit 2.5.4 upgrade
In-Reply-To: <51C7471C.70504@redhat.com>
References: <51C7471C.70504@redhat.com>
Message-ID: <673626329.1367730.1372064280345.JavaMail.root@redhat.com>



----- Original Message -----
> From: "Itamar Heim" <iheim at redhat.com>
> To: "arch" <arch at ovirt.org>
> Sent: Sunday, June 23, 2013 10:06:04 PM
> Subject: Gerrit 2.5.4 upgrade
> 
> just to sum up the highlights of upgrade from June 9th:
> 
> - Gerrit was upgraded to 2.5.4
> 
> - new REST API
> http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.5.html#_rest_api
> 
> - Custom Dashboards
> http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/user-custom-dashboards.html
> 
> - the inline email are now part of gerrit (we used to run a patched 2.4
>    version with them)
> 
> - new prolog submit rules (still considering how can be best used)
> http://gerrit-documentation.googlecode.com/svn/Documentation/2.5/prolog-cookbook.html
My initial instinct is that this can be used to enforce "soft" policies we currently have in place.
E.g., by policy, we have different engine-core, engine-fronted and rest maintainers.
However, technically, these are all part of the oVirt Engine project, and, e.g., a core maintainer could give +2 and merge a patch that contains rest/frontend changes.
Seems like the problog submit rules can help overcome this.

> 
> - the mirroring plugin (to github.com/oVirt) is now a plugin (Thanks
>    David)
> 
> - looking forward to gerrit 2.6 upgrade which promises "42x improvement
>    on git clone and git fetch"
Please, please, upgrade to 2.6 :-)

> 
> (I also remember it was supposed to improve copying approval flags on
> rebase, but couldn't find the reference).
> 
> Thanks,
>     Itamar
> _______________________________________________
> Arch mailing list
> Arch at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/arch
> 


From iheim at redhat.com  Mon Jun 24 09:13:17 2013
From: iheim at redhat.com (Itamar Heim)
Date: Mon, 24 Jun 2013 12:13:17 +0300
Subject: Gerrit 2.5.4 upgrade
In-Reply-To: <673626329.1367730.1372064280345.JavaMail.root@redhat.com>
References: <51C7471C.70504@redhat.com>
	<673626329.1367730.1372064280345.JavaMail.root@redhat.com>
Message-ID: <51C80DAD.6010509@redhat.com>

On 06/24/2013 11:58 AM, Allon Mureinik wrote:
>> >
>> >- the mirroring plugin (to github.com/oVirt) is now a plugin (Thanks
>> >    David)
>> >
>> >- looking forward to gerrit 2.6 upgrade which promises "42x improvement
>> >    on git clone and git fetch"
> Please, please, upgrade to 2.6:-)
>

well, it was just released last Friday, i'm waiting a few weeks before 
doing it.


From deepakcs at linux.vnet.ibm.com  Tue Jun 25 14:23:13 2013
From: deepakcs at linux.vnet.ibm.com (Deepak C Shetty)
Date: Tue, 25 Jun 2013 19:53:13 +0530
Subject: Few questions on Backup Restore API implmn.
In-Reply-To: <1944585584.16867066.1371472040674.JavaMail.root@redhat.com>
References: <51A76B36.7030408@linux.vnet.ibm.com> <51AF12B2.3000903@redhat.com>
	<51AF1DA2.5070305@linux.vnet.ibm.com> <51AF257D.6040800@redhat.com>
	<350995831.12892634.1370560204679.JavaMail.root@redhat.com>
	<51B43337.5000104@redhat.com>
	<1433156354.14066468.1370903923309.JavaMail.root@redhat.com>
	<51BD9377.7070405@redhat.com>
	<1944585584.16867066.1371472040674.JavaMail.root@redhat.com>
Message-ID: <51C9A7D1.7020601@linux.vnet.ibm.com>

On 06/17/2013 05:57 PM, Ayal Baron wrote:
>
> ----- Original Message -----
>> Hi Ayal,
>>
>> On 06/11/2013 01:38 AM, Ayal Baron wrote:
>>> I think there is some misunderstanding here.
>>> The 'users' *are* the backup vendors.
>>> These APIs are intended directly for backup vendors who will integrate once
>>> with the API.
>>> Users wanting to backup or restore will go to the vendors application not
>>> to oVirt.
>> Well this is whole different story, i could never guess that from the feature
>> wiki,
>> obviously when speaking about public api, first thing that pops up is a ovirt
>> user,
>> not backup vendor.
>>
>> So AFAIU the backup process will be triggered/managed from the backup vendor
>> app.,
>> in that case i'm perfectly fine with the current design.
> Ack, so we'll fix the wiki to explain this better, sorry for the confusion, I had a feeling we're not talking about the same things here and I do agree that for an oVirt user rather than vendor it would be a pain in the $%@#.

I changed the 'Summary' on the wiki page from ...

"This feature allows oVirt users to backup and restore VMs using 
Independent Software Vendors. New set of APIs will be introduced in 
oVirt to facilitate taking full VM backup, as well as full or file level 
restore of VMs."

to ...

"This feature provides the ability for ISVs to backup and restore VMs. 
New set of APIs will be introduced in oVirt to facilitate taking full VM 
backup, as well as full or file level restore of VMs. Backup and Restore 
will be REST API driven (and not GUI/User driven)."

Other than that.. i didn't see any other text which will cause the 
confusion.

thanx,
deepak


>
>
>>
>> cheers.
>>
>> --
>>
>> Michael Pasternak
>> RedHat, ENG-Virtualization R&D
>>
>
>



From agl at us.ibm.com  Tue Jun 25 15:23:51 2013
From: agl at us.ibm.com (Adam Litke)
Date: Tue, 25 Jun 2013 11:23:51 -0400
Subject: encounter problem to detect IO bandwidth congestion in ovirt
	storage domains
In-Reply-To: <51BFDB47.3040908@linux.vnet.ibm.com>
References: <51BFDB47.3040908@linux.vnet.ibm.com>
Message-ID: <20130625152351.GT3484@aglitke.home>

On Tue, Jun 18, 2013 at 12:00:07PM +0800, Mei EL Liu wrote:
> Hi,
> 
> I found that its a little difficult to detect IO bandwidth
> congestion in ovirt storage domains supported by NFS or GlusterFs.
> 
> For block based storage, it's easier to detect, since you can use
> some tool like iostat.For the case of file system based storage,
> it's much harder.
> 
> I investigate the existing solution. vsphere uses average IO latency
> to detect it. I propose a similar scheme in
> http://www.ovirt.org/Features/Design/SLA_for_storage_io_bandwidth .
> It simplifies the scheme by make the congestion decision on a single
> host instead of using the statistics from all the hosts use the
> backend storage. It doesn't need communication between hosts and
> maybe in phase two we can add communication and make a global
> decision.
> 
>  For now, it detects congestion viastatistics of vms using that
> backend storage in the local host(This info is collected through
> iostat in vm).  It collects IO latency in such vms and compute an
> average latency for that backend storage. If it is higher than
> threshold, a congestion is found.
> 
> However, when I did testing for such a policy, I found that setting
> IO limit to a smaller value will make latency longer. That means if
> average latency exceeds that threshold and then our automatically
> tuning IO limit will be decreased which lead to average IO latency
> longer. Of course, this IO latency will exceed the threshold again
> and cause the IO limit be decreased. This will finally cause the IO
> limit to its lower bound.
> 
> This scheme has affected by the following reasons:
> 1. we collect statistic data from vms instead of host. (This is
> because it is hard to collect such info for remote storage like NFS,
> GlusterFS)
> 2.The IO limit affect the latency.
> 3. The threshold is a constant.
> 4 I also find that iostat's await(latency info) is not good enough
> since the latency is long for very light io or very heavy IO.
> 
> 
> Does anybody get an idea or have experience on this? Suggestions are
> more than welcome. Thanks in advance.

Thank you for such a thorough introduction to this topic.

One thought I had is that maybe we need to invert your logic with
respect to IO throttling.  The way it could work is:

1) At the datacenter level, we establish a throughput range.  VMs are
   guaranteed the minimum and won't exceed the maximum.  Similarly, we
   set a range for latency.

2) Hosts continuously reduce allowable bandwidth for VMs down to the
   throughput minimum.  If latency rises above the allowable limit for a
   single VM slowly increase bandwidth up to allowable maximum.

3) If over time, the IO remains congested you can:
   a) Decrease the cluster-wide throughput minimum and maximum values
   b) Increase the maximum allowable latency
   c) Migrate VM disks to alternate storage
   d) Upgrade Storage


-- 
Adam Litke <agl at us.ibm.com>
IBM Linux Technology Center