Hi all, I would like propose adding the Memory Overcommitment Manager (MOM) project to oVirt. MOM is a daemon that is used to apply dynamic host-level and VM-level tuning according to a customizable policy. Today MOM can control memory ballooning and KSM. Please see the following links for more information: Project wiki: http://github.com/aglitke/mom/wiki Source code: git://github.com/aglitke/mom.git KVM Forum presentation and video: http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf http://vimeo.com/15223652 developerWorks article: http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ In my view, MOM could be nicely integrated with oVirt-node and VDSM. I am looking forward to your comments and am ready to address some next steps. Thanks!

On 09/26/2011 03:19 PM, Dor Laor wrote: > On 09/26/2011 11:01 PM, Adam Litke wrote: >> Hi all, >> >> I would like propose adding the Memory Overcommitment Manager (MOM) >> project to >> oVirt. MOM is a daemon that is used to apply dynamic host-level and >> VM-level >> tuning according to a customizable policy. Today MOM can control memory >> ballooning and KSM. Please see the following links for more >> information: >> >> Project wiki: >> http://github.com/aglitke/mom/wiki >> >> Source code: >> git://github.com/aglitke/mom.git >> >> KVM Forum presentation and video: >> http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf >> >> http://vimeo.com/15223652 >> >> developerWorks article: >> http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ >> >> >> In my view, MOM could be nicely integrated with oVirt-node and VDSM. >> I am >> looking forward to your comments and am ready to address some next >> steps. >> >> Thanks! >> > > +1 for mom, hope to see a dad too :) Data Acceleration Director? Joking aside, would MOM just be integrated directly into VDSM or would it remain as a separate project? I can envision a world where MOM is just a feature of VDSM.

----- Original Message ----- > On 09/26/2011 04:26 PM, Anthony Liguori wrote: > > On 09/26/2011 03:19 PM, Dor Laor wrote: > >> On 09/26/2011 11:01 PM, Adam Litke wrote: > >>> Hi all, > >>> > >>> I would like propose adding the Memory Overcommitment Manager > >>> (MOM) > >>> project to > >>> oVirt. MOM is a daemon that is used to apply dynamic host-level > >>> and > >>> VM-level > >>> tuning according to a customizable policy. Today MOM can control > >>> memory > >>> ballooning and KSM. Please see the following links for more > >>> information: > >>> > >>> Project wiki: > >>> http://github.com/aglitke/mom/wiki > >>> > >>> Source code: > >>> git://github.com/aglitke/mom.git > >>> > >>> KVM Forum presentation and video: > >>> http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf > >>> > >>> http://vimeo.com/15223652 > >>> > >>> developerWorks article: > >>> http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ > >>> > >>> > >>> In my view, MOM could be nicely integrated with oVirt-node and > >>> VDSM. > >>> I am > >>> looking forward to your comments and am ready to address some > >>> next > >>> steps. > >>> > >>> Thanks! > >>> > >> > >> +1 for mom, hope to see a dad too :) > > > > Data Acceleration Director? > > > > Joking aside, would MOM just be integrated directly into VDSM or > > would > > it remain as a separate project? > > > > I can envision a world where MOM is just a feature of VDSM. > > The key question is how do we integrate it into the stack, > - is it integrated into WDSM as Anthony mentions > - or is a project that WDSM uses to provide MOM > - etc. First of all I think this is great! Thanks Adam! Wrt the question above, the way I see vdsm going forward is a lot more policy management, especially once we start talking about SLAs. This will involve a lot more than just memory policies, and although I may be impartial here, I think that this should fall under vdsm or under a policy engine project (bigger scope than just memory) which vdsm would use. To elaborate, in addition to memory, we need to manage policies around network, storage and cpu and manipulate cgroups, tc, page cache etc. If we take page cache as an example, it affects memory on the machine hence, MOM would like to tweak it (also in the preso), but we already have recent results from the perf team to change the defaults here to significantly increase I/O throughput, improve fairness and reduce effect of buffered writers on VMs and we know that going forward we'll have to tweak these parameters dynamically to adjust to varying workloads, esp if we ever plan on making good the claim for a hybrid server. Having separate projects tweaking the same params will not lead to good results.

On 09/27/2011 12:52 AM, Adam Litke wrote: >> > First of all I think this is great! Thanks Adam! >> > Wrt the question above, the way I see vdsm going forward is a >> > lot more policy >> > management, especially once we start talking about SLAs. This >> > will involve a >> > lot more than just memory policies, and although I may be >> > impartial here, I >> > think that this should fall under vdsm or under a policy engine >> > project >> > (bigger scope than just memory) which vdsm would use. To >> > elaborate, in >> > addition to memory, we need to manage policies around network, >> > storage and cpu >> > and manipulate cgroups, tc, page cache etc. If we take page >> > cache as an >> > example, it affects memory on the machine hence, MOM would like >> > to tweak it >> > (also in the preso), but we already have recent results from >> > the perf team to >> > change the defaults here to significantly increase I/O >> > throughput, improve >> > fairness and reduce effect of buffered writers on VMs and we >> > know that going >> > forward we'll have to tweak these parameters dynamically to >> > adjust to varying >> > workloads, esp if we ever plan on making good the claim for a >> > hybrid server. >> > Having separate projects tweaking the same params will not lead >> > to good >> > results. > Ayal, I definitely agree. I must admit that the project name MOM > (while a nice > acronym), implies a restricted scope (memory only) but the > architecture of MOM > does not limit its functionality to memory tuning only. I assert > that MOM can > easily become the policy engine piece that you are speaking of. > > The framework is very extensible (a set of Collectors that expose > system state, > a set of Controllers that expose "tuning knobs", and a policy that > reads > the state and manipulates the knobs. Adding control for cgroups is > as simple as > writing a cgroups Collector and Controller. It would be nice to agree on the initial (more limited scope maybe) for the code within vdsm so the work be able to be done soonish. Since vdsm is a relevantly a new open source project there will be many future changes and imho there is no need to wait for the perfect policy engine framework.

From: "Alexander Graf" <agraf(a)suse.de&gt; To: "Ayal Baron" <abaron(a)redhat.com&gt; Cc: &quot;&lt;board(a)ovirt.org&gt;&quot; <board(a)ovirt.org&gt; Sent: Tuesday, September 27, 2011 3:28:29 AM Subject: Re: Adding Memory Overcommitment Manager (MOM) to oVirt Am 27.09.2011 um 08:56 schrieb "Ayal Baron" <abaron(a)redhat.com&gt;: > > > ----- Original Message ----- >> On 09/27/2011 12:52 AM, Adam Litke wrote: >>>>> First of all I think this is great! Thanks Adam! >>>>> Wrt the question above, the way I see vdsm going forward is a >>>>> lot more policy >>>>> management, especially once we start talking about SLAs. This >>>>> will involve a >>>>> lot more than just memory policies, and although I may be >>>>> impartial here, I >>>>> think that this should fall under vdsm or under a policy engine >>>>> project >>>>> (bigger scope than just memory) which vdsm would use. To >>>>> elaborate, in >>>>> addition to memory, we need to manage policies around network, >>>>> storage and cpu >>>>> and manipulate cgroups, tc, page cache etc. If we take page >>>>> cache as an >>>>> example, it affects memory on the machine hence, MOM would like >>>>> to tweak it >>>>> (also in the preso), but we already have recent results from >>>>> the perf team to >>>>> change the defaults here to significantly increase I/O >>>>> throughput, improve >>>>> fairness and reduce effect of buffered writers on VMs and we >>>>> know that going >>>>> forward we'll have to tweak these parameters dynamically to >>>>> adjust to varying >>>>> workloads, esp if we ever plan on making good the claim for a >>>>> hybrid server. >>>>> Having separate projects tweaking the same params will not lead >>>>> to good >>>>> results. >>> Ayal, I definitely agree. I must admit that the project name MOM >>> (while a nice >>> acronym), implies a restricted scope (memory only) but the >>> architecture of MOM >>> does not limit its functionality to memory tuning only. I assert >>> that MOM can >>> easily become the policy engine piece that you are speaking of. >>> >>> The framework is very extensible (a set of Collectors that expose >>> system state, >>> a set of Controllers that expose "tuning knobs", and a policy >>> that >>> reads >>> the state and manipulates the knobs. Adding control for cgroups >>> is >>> as simple as >>> writing a cgroups Collector and Controller. >> >> It would be nice to agree on the initial (more limited scope >> maybe) >> for >> the code within vdsm so the work be able to be done soonish. Since >> vdsm >> is a relevantly a new open source project there will be many >> future >> changes and imho there is no need to wait for the perfect policy >> engine >> framework. >> > > The project can initially contain only memory management, that is > not an issue whatsoever. > But the project mission should be correct from the start so that > design decisions are made with the full picture in mind. Let me try to grasp the full picture here. VDSM is basically the 'node controller', managing all active VMs on a system, correct? So this is the instance that watches things on node granularity. MOM watches resources on node granularity, interacting only with vdsm and the VM(?) to manage overcommit of memory. Why should this be separate from VDSM? Does it buy us anything as a separate project? Is the interface cut here helpful in any way? I don't want to see 50 separate daemons that in reality are a single entity but have to be artificially maintained separate, because someone thought it's nice to split things into projects instead of submodules of a project. So what's the benefit of a separate MOM vs an integrated MOM inside of VDSM?

Alex _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

> > So what's the benefit of a separate MOM vs an integrated MOM inside > > of VDSM? > > I'd agree this is really VM policy which should be handled by VDSM. It's pretty simple. MOM does something valuable today that VDSM doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM wants to incorporate MOM that's great. But how is the existence of VDSM an argument against contributing MOM source code to the oVirt community? Am I missing something? On a related point, there are good examples that argue for consolidating function inside a single daemon, and there are good counterexamples. It's not always true that every node policy function should be integrated within a single daemon. It probably makes sense in this case. But again, VDSM doesn't do what MOM does today, which is another argument in favor of contributing MOM and letting the community work with it.

On Tue, Sep 27, 2011 at 09:53:00AM -0400, Carl Trieloff wrote: > On 09/27/2011 09:43 AM, Michael D Day wrote: >>>> So what's the benefit of a separate MOM vs an integrated MOM inside >>>> of VDSM? >>> I'd agree this is really VM policy which should be handled by VDSM. >> It's pretty simple. MOM does something valuable today that VDSM >> doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM >> wants to incorporate MOM that's great. But how is the existence of >> VDSM an argument against contributing MOM source code to the oVirt >> community? Am I missing something? >> >> On a related point, there are good examples that argue for >> consolidating function inside a single daemon, and there are good >> counterexamples. It's not always true that every node policy function >> should be integrated within a single daemon. It probably makes sense >> in this case. But again, VDSM doesn't do what MOM does today, which is >> another argument in favor of contributing MOM and letting the >> community work with it. > > I think the debate has ascertained that we want it. The question is how > we integrate it... (used by VDSM or additional daemon) I like Dor's > suggestion. Let's let the guys figure out the best way to do an initial > integration given where we are at, we play with it, and if in future we > want to evolve the integration we can. I completely agree and (boldly perhaps) assume we are driving at a consensus here. In that case, what are the next steps for MOM from an oVirt project perspective? For Hosting, the project happily resides on github. I created a mailing list using Google Groups, but discussions have mostly been had on other projects' lists. Although I am not completely opposed to renaming it, I am not sure it's worth the effort if we can see uses for this project outside of the oVirt umbrella. Perhaps we can revisit that in the future as things shake out. A decent next step would be to get MOM included into Fedora. I have a stalled request that I could use some help moving along: https://bugzilla.redhat.com/show_bug.cgi?id=638647 The package is ready (spec file has been approved) but needs a Fedora package sponsor who is willing to guide it through the remaining steps of the process. Anything else?

From: "Michael D Day" <mdday(a)us.ibm.com&gt; To: "Andrew Cathrow" <acathrow(a)redhat.com&gt; Cc: "Alexander Graf" <agraf(a)suse.de&gt;, "board" <board(a)ovirt.org&gt;, board-bounces(a)ovirt.org Sent: Tuesday, September 27, 2011 9:43:39 AM Subject: Re: Adding Memory Overcommitment Manager (MOM) to oVirt > > So what's the benefit of a separate MOM vs an integrated MOM > > inside > > of VDSM? > > I'd agree this is really VM policy which should be handled by VDSM. It's pretty simple. MOM does something valuable today that VDSM doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM wants to incorporate MOM that's great. But how is the existence of VDSM an argument against contributing MOM source code to the oVirt community? Am I missing something?

On a related point, there are good examples that argue for consolidating function inside a single daemon, and there are good counterexamples. It's not always true that every node policy function should be integrated within a single daemon. It probably makes sense in this case. But again, VDSM doesn't do what MOM does today, which is another argument in favor of contributing MOM and letting the community work with it. Thanks, Mike Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday(a)us.ibm.com http://code.ncultra.org

I think this discussion highlights the need for a bit more structure for adding a sub project. I would like to propose the following: 1) Project posts an overview page, including scope and short term roadmap to ovirt.org wiki. That page is then copied as a mail to board@ovirt 2) There is an official comment period that lasts for a fixed period of time (2-4 weeks?). This time period allows for questions to be enhanced, and the wiki to be improved. 3) At any point in this time period, the proposal can be officially withdrawn. 4) At the end of the period, there is a final call for comments before a vote takes place.

The other thing I was thinking is that could have Acked-by's and wait to get N Acked-by's before going to a vote. The reason for having a time period was to give people time to actually look at code and read proposals instead of just immediately jumping into a heated debate. I think forcing a little think time is a good thing.

> > Let's also add the basic questions for the new project to answer, and > turn this into an additional page on the site. Do we have a wiki ETA? I'd be happy to start putting together questions.

On Tue, Sep 27, 2011 at 11:03:01AM -0400, Carl Trieloff wrote: > wiki is live. > > ovirt.org/wiki Great! I would like to create a page to discuss MOM oVirt integration. Could I have an account from an existing account holder (username: aglitke) please? Thanks!

-- Adam Litke <agl(a)us.ibm.com&gt; IBM Linux Technology Center _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

> > So what's the benefit of a separate MOM vs an integrated MOM > > inside > > of VDSM? > > I'd agree this is really VM policy which should be handled by VDSM. It's pretty simple. MOM does something valuable today that VDSM doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM wants to incorporate MOM that's great. But how is the existence of VDSM an argument against contributing MOM source code to the oVirt community? Am I missing something?

On a related point, there are good examples that argue for consolidating function inside a single daemon, and there are good counterexamples. It's not always true that every node policy function should be integrated within a single daemon. It probably makes sense in this case. But again, VDSM doesn't do what MOM does today, which is another argument in favor of contributing MOM and letting the community work with it.

Thanks, Mike Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday(a)us.ibm.com http://code.ncultra.org _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

Ayal Baron <abaron(a)redhat.com&gt; wrote on 09/27/2011 10:02:23 AM: > No one is against accepting MOM into oVirt, on the contrary, we > really want it. > The only question is do we want to assimilate the code into vdsm or > manage it as a separate project. Thanks for the clarification folks, I read in-between the lines of Andy's single-line response :-).

On Mon, Sep 26, 2011 at 05:28:54PM -0400, Ayal Baron wrote: > > > ----- Original Message ----- > > On 09/26/2011 04:26 PM, Anthony Liguori wrote: > > > On 09/26/2011 03:19 PM, Dor Laor wrote: > > >> On 09/26/2011 11:01 PM, Adam Litke wrote: > > >>> Hi all, > > >>> > > >>> I would like propose adding the Memory Overcommitment Manager > > >>> (MOM) > > >>> project to > > >>> oVirt. MOM is a daemon that is used to apply dynamic > > >>> host-level > > >>> and > > >>> VM-level > > >>> tuning according to a customizable policy. Today MOM can > > >>> control > > >>> memory > > >>> ballooning and KSM. Please see the following links for more > > >>> information: > > >>> > > >>> Project wiki: > > >>> http://github.com/aglitke/mom/wiki > > >>> > > >>> Source code: > > >>> git://github.com/aglitke/mom.git > > >>> > > >>> KVM Forum presentation and video: > > >>> http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf > > >>> > > >>> http://vimeo.com/15223652 > > >>> > > >>> developerWorks article: > > >>> http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ > > >>> > > >>> > > >>> In my view, MOM could be nicely integrated with oVirt-node > > >>> and > > >>> VDSM. > > >>> I am > > >>> looking forward to your comments and am ready to address some > > >>> next > > >>> steps. > > >>> > > >>> Thanks! > > >>> > > >> > > >> +1 for mom, hope to see a dad too :) > > > > > > Data Acceleration Director? > > > > > > Joking aside, would MOM just be integrated directly into VDSM > > > or > > > would > > > it remain as a separate project? > > > > > > I can envision a world where MOM is just a feature of VDSM. > > > > The key question is how do we integrate it into the stack, > > - is it integrated into WDSM as Anthony mentions > > - or is a project that WDSM uses to provide MOM > > - etc. > > First of all I think this is great! Thanks Adam! > Wrt the question above, the way I see vdsm going forward is a lot > more policy > management, especially once we start talking about SLAs. This will > involve a > lot more than just memory policies, and although I may be impartial > here, I > think that this should fall under vdsm or under a policy engine > project > (bigger scope than just memory) which vdsm would use. To > elaborate, in > addition to memory, we need to manage policies around network, > storage and cpu > and manipulate cgroups, tc, page cache etc. If we take page cache > as an > example, it affects memory on the machine hence, MOM would like to > tweak it > (also in the preso), but we already have recent results from the > perf team to > change the defaults here to significantly increase I/O throughput, > improve > fairness and reduce effect of buffered writers on VMs and we know > that going > forward we'll have to tweak these parameters dynamically to adjust > to varying > workloads, esp if we ever plan on making good the claim for a > hybrid server. > Having separate projects tweaking the same params will not lead to > good > results. Ayal, I definitely agree. I must admit that the project name MOM (while a nice acronym), implies a restricted scope (memory only) but the architecture of MOM does not limit its functionality to memory tuning only. I assert that MOM can easily become the policy engine piece that you are speaking of. The framework is very extensible (a set of Collectors that expose system state, a set of Controllers that expose "tuning knobs", and a policy that reads the state and manipulates the knobs. Adding control for cgroups is as simple as writing a cgroups Collector and Controller.

-- Adam Litke <agl(a)us.ibm.com&gt; IBM Linux Technology Center