Hi all, I would like propose adding the Memory Overcommitment Manager (MOM) project to oVirt. MOM is a daemon that is used to apply dynamic host-level and VM-level tuning according to a customizable policy. Today MOM can control memory ballooning and KSM. Please see the following links for more information: Project wiki: http://github.com/aglitke/mom/wiki Source code: git://github.com/aglitke/mom.git KVM Forum presentation and video: http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf http://vimeo.com/15223652 developerWorks article: http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ In my view, MOM could be nicely integrated with oVirt-node and VDSM. I am looking forward to your comments and am ready to address some next steps. Thanks!

On 09/26/2011 03:19 PM, Dor Laor wrote: > On 09/26/2011 11:01 PM, Adam Litke wrote: >> Hi all, >> >> I would like propose adding the Memory Overcommitment Manager (MOM) >> project to >> oVirt. MOM is a daemon that is used to apply dynamic host-level and >> VM-level >> tuning according to a customizable policy. Today MOM can control memory >> ballooning and KSM. Please see the following links for more >> information: >> >> Project wiki: >> http://github.com/aglitke/mom/wiki >> >> Source code: >> git://github.com/aglitke/mom.git >> >> KVM Forum presentation and video: >> http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf >> >> http://vimeo.com/15223652 >> >> developerWorks article: >> http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ >> >> >> In my view, MOM could be nicely integrated with oVirt-node and VDSM. >> I am >> looking forward to your comments and am ready to address some next >> steps. >> >> Thanks! >> > > +1 for mom, hope to see a dad too :) Data Acceleration Director? Joking aside, would MOM just be integrated directly into VDSM or would it remain as a separate project? I can envision a world where MOM is just a feature of VDSM.

----- Original Message ----- > On 09/26/2011 04:26 PM, Anthony Liguori wrote: > > On 09/26/2011 03:19 PM, Dor Laor wrote: > >> On 09/26/2011 11:01 PM, Adam Litke wrote: > >>> Hi all, > >>> > >>> I would like propose adding the Memory Overcommitment Manager > >>> (MOM) > >>> project to > >>> oVirt. MOM is a daemon that is used to apply dynamic host-level > >>> and > >>> VM-level > >>> tuning according to a customizable policy. Today MOM can control > >>> memory > >>> ballooning and KSM. Please see the following links for more > >>> information: > >>> > >>> Project wiki: > >>> http://github.com/aglitke/mom/wiki > >>> > >>> Source code: > >>> git://github.com/aglitke/mom.git > >>> > >>> KVM Forum presentation and video: > >>> http://www.linux-kvm.org/wiki/images/e/e8/2010-forum-litke-kvmforum2010.pdf > >>> > >>> http://vimeo.com/15223652 > >>> > >>> developerWorks article: > >>> http://www.ibm.com/developerworks/linux/library/l-overcommit-kvm-resources/ > >>> > >>> > >>> In my view, MOM could be nicely integrated with oVirt-node and > >>> VDSM. > >>> I am > >>> looking forward to your comments and am ready to address some > >>> next > >>> steps. > >>> > >>> Thanks! > >>> > >> > >> +1 for mom, hope to see a dad too :) > > > > Data Acceleration Director? > > > > Joking aside, would MOM just be integrated directly into VDSM or > > would > > it remain as a separate project? > > > > I can envision a world where MOM is just a feature of VDSM. > > The key question is how do we integrate it into the stack, > - is it integrated into WDSM as Anthony mentions > - or is a project that WDSM uses to provide MOM > - etc. First of all I think this is great! Thanks Adam! Wrt the question above, the way I see vdsm going forward is a lot more policy management, especially once we start talking about SLAs. This will involve a lot more than just memory policies, and although I may be impartial here, I think that this should fall under vdsm or under a policy engine project (bigger scope than just memory) which vdsm would use. To elaborate, in addition to memory, we need to manage policies around network, storage and cpu and manipulate cgroups, tc, page cache etc. If we take page cache as an example, it affects memory on the machine hence, MOM would like to tweak it (also in the preso), but we already have recent results from the perf team to change the defaults here to significantly increase I/O throughput, improve fairness and reduce effect of buffered writers on VMs and we know that going forward we'll have to tweak these parameters dynamically to adjust to varying workloads, esp if we ever plan on making good the claim for a hybrid server. Having separate projects tweaking the same params will not lead to good results.

> > First of all I think this is great! Thanks Adam! > > Wrt the question above, the way I see vdsm going forward is a lot more policy > > management, especially once we start talking about SLAs. This will involve a > > lot more than just memory policies, and although I may be impartial here, I > > think that this should fall under vdsm or under a policy engine project > > (bigger scope than just memory) which vdsm would use. To elaborate, in > > addition to memory, we need to manage policies around network, storage and cpu > > and manipulate cgroups, tc, page cache etc. If we take page cache as an > > example, it affects memory on the machine hence, MOM would like to tweak it > > (also in the preso), but we already have recent results from the perf team to > > change the defaults here to significantly increase I/O throughput, improve > > fairness and reduce effect of buffered writers on VMs and we know that going > > forward we'll have to tweak these parameters dynamically to adjust to varying > > workloads, esp if we ever plan on making good the claim for a hybrid server. > > Having separate projects tweaking the same params will not lead to good > > results. Ayal, I definitely agree. I must admit that the project name MOM (while a nice acronym), implies a restricted scope (memory only) but the architecture of MOM does not limit its functionality to memory tuning only. I assert that MOM can easily become the policy engine piece that you are speaking of. The framework is very extensible (a set of Collectors that expose system state, a set of Controllers that expose "tuning knobs", and a policy that reads the state and manipulates the knobs. Adding control for cgroups is as simple as writing a cgroups Collector and Controller.

On 09/27/2011 12:52 AM, Adam Litke wrote: >> > First of all I think this is great! Thanks Adam! >> > Wrt the question above, the way I see vdsm going forward is a >> > lot more policy >> > management, especially once we start talking about SLAs. This >> > will involve a >> > lot more than just memory policies, and although I may be >> > impartial here, I >> > think that this should fall under vdsm or under a policy engine >> > project >> > (bigger scope than just memory) which vdsm would use. To >> > elaborate, in >> > addition to memory, we need to manage policies around network, >> > storage and cpu >> > and manipulate cgroups, tc, page cache etc. If we take page >> > cache as an >> > example, it affects memory on the machine hence, MOM would like >> > to tweak it >> > (also in the preso), but we already have recent results from >> > the perf team to >> > change the defaults here to significantly increase I/O >> > throughput, improve >> > fairness and reduce effect of buffered writers on VMs and we >> > know that going >> > forward we'll have to tweak these parameters dynamically to >> > adjust to varying >> > workloads, esp if we ever plan on making good the claim for a >> > hybrid server. >> > Having separate projects tweaking the same params will not lead >> > to good >> > results. > Ayal, I definitely agree. I must admit that the project name MOM > (while a nice > acronym), implies a restricted scope (memory only) but the > architecture of MOM > does not limit its functionality to memory tuning only. I assert > that MOM can > easily become the policy engine piece that you are speaking of. > > The framework is very extensible (a set of Collectors that expose > system state, > a set of Controllers that expose "tuning knobs", and a policy that > reads > the state and manipulates the knobs. Adding control for cgroups is > as simple as > writing a cgroups Collector and Controller. It would be nice to agree on the initial (more limited scope maybe) for the code within vdsm so the work be able to be done soonish. Since vdsm is a relevantly a new open source project there will be many future changes and imho there is no need to wait for the perfect policy engine framework.

From: "Alexander Graf" <agraf(a)suse.de&gt; To: "Ayal Baron" <abaron(a)redhat.com&gt; Cc: &quot;&lt;board(a)ovirt.org&gt;&quot; <board(a)ovirt.org&gt; Sent: Tuesday, September 27, 2011 3:28:29 AM Subject: Re: Adding Memory Overcommitment Manager (MOM) to oVirt Am 27.09.2011 um 08:56 schrieb "Ayal Baron" <abaron(a)redhat.com&gt;: > > > ----- Original Message ----- >> On 09/27/2011 12:52 AM, Adam Litke wrote: >>>>> First of all I think this is great! Thanks Adam! >>>>> Wrt the question above, the way I see vdsm going forward is a >>>>> lot more policy >>>>> management, especially once we start talking about SLAs. This >>>>> will involve a >>>>> lot more than just memory policies, and although I may be >>>>> impartial here, I >>>>> think that this should fall under vdsm or under a policy engine >>>>> project >>>>> (bigger scope than just memory) which vdsm would use. To >>>>> elaborate, in >>>>> addition to memory, we need to manage policies around network, >>>>> storage and cpu >>>>> and manipulate cgroups, tc, page cache etc. If we take page >>>>> cache as an >>>>> example, it affects memory on the machine hence, MOM would like >>>>> to tweak it >>>>> (also in the preso), but we already have recent results from >>>>> the perf team to >>>>> change the defaults here to significantly increase I/O >>>>> throughput, improve >>>>> fairness and reduce effect of buffered writers on VMs and we >>>>> know that going >>>>> forward we'll have to tweak these parameters dynamically to >>>>> adjust to varying >>>>> workloads, esp if we ever plan on making good the claim for a >>>>> hybrid server. >>>>> Having separate projects tweaking the same params will not lead >>>>> to good >>>>> results. >>> Ayal, I definitely agree. I must admit that the project name MOM >>> (while a nice >>> acronym), implies a restricted scope (memory only) but the >>> architecture of MOM >>> does not limit its functionality to memory tuning only. I assert >>> that MOM can >>> easily become the policy engine piece that you are speaking of. >>> >>> The framework is very extensible (a set of Collectors that expose >>> system state, >>> a set of Controllers that expose "tuning knobs", and a policy >>> that >>> reads >>> the state and manipulates the knobs. Adding control for cgroups >>> is >>> as simple as >>> writing a cgroups Collector and Controller. >> >> It would be nice to agree on the initial (more limited scope >> maybe) >> for >> the code within vdsm so the work be able to be done soonish. Since >> vdsm >> is a relevantly a new open source project there will be many >> future >> changes and imho there is no need to wait for the perfect policy >> engine >> framework. >> > > The project can initially contain only memory management, that is > not an issue whatsoever. > But the project mission should be correct from the start so that > design decisions are made with the full picture in mind. Let me try to grasp the full picture here. VDSM is basically the 'node controller', managing all active VMs on a system, correct? So this is the instance that watches things on node granularity. MOM watches resources on node granularity, interacting only with vdsm and the VM(?) to manage overcommit of memory. Why should this be separate from VDSM? Does it buy us anything as a separate project? Is the interface cut here helpful in any way? I don't want to see 50 separate daemons that in reality are a single entity but have to be artificially maintained separate, because someone thought it's nice to split things into projects instead of submodules of a project. So what's the benefit of a separate MOM vs an integrated MOM inside of VDSM?

Alex _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

> > So what's the benefit of a separate MOM vs an integrated MOM inside > > of VDSM? > > I'd agree this is really VM policy which should be handled by VDSM. It's pretty simple. MOM does something valuable today that VDSM doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM wants to incorporate MOM that's great. But how is the existence of VDSM an argument against contributing MOM source code to the oVirt community? Am I missing something? On a related point, there are good examples that argue for consolidating function inside a single daemon, and there are good counterexamples. It's not always true that every node policy function should be integrated within a single daemon. It probably makes sense in this case. But again, VDSM doesn't do what MOM does today, which is another argument in favor of contributing MOM and letting the community work with it.

From: "Michael D Day" <mdday(a)us.ibm.com&gt; To: "Andrew Cathrow" <acathrow(a)redhat.com&gt; Cc: "Alexander Graf" <agraf(a)suse.de&gt;, "board" <board(a)ovirt.org&gt;, board-bounces(a)ovirt.org Sent: Tuesday, September 27, 2011 9:43:39 AM Subject: Re: Adding Memory Overcommitment Manager (MOM) to oVirt > > So what's the benefit of a separate MOM vs an integrated MOM > > inside > > of VDSM? > > I'd agree this is really VM policy which should be handled by VDSM. It's pretty simple. MOM does something valuable today that VDSM doesn't do. An integrated MOM inside of VDSM doesn't exist. If VDSM wants to incorporate MOM that's great. But how is the existence of VDSM an argument against contributing MOM source code to the oVirt community? Am I missing something?

On a related point, there are good examples that argue for consolidating function inside a single daemon, and there are good counterexamples. It's not always true that every node policy function should be integrated within a single daemon. It probably makes sense in this case. But again, VDSM doesn't do what MOM does today, which is another argument in favor of contributing MOM and letting the community work with it. Thanks, Mike Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday(a)us.ibm.com http://code.ncultra.org

No one is against accepting MOM into oVirt, on the contrary, we really want it. The only question is do we want to assimilate the code into vdsm or manage it as a separate project.

Sorry, I should have given a multi line reply, I was thinking a few steps ahead. Bringing MOM under the oVirt umbrella would be a great thing - whether that's as a standalone project or as part of VDSM. Personally I believe it will/should end up as part of VDSM, but either way - if VDSM leverages MOM (like it leverages libvirt) or if it gets merged into VDSM this should happen under the oVirt umbrella. Adding MOM to oVirt would be great and also helps to show momentum, maybe it remains standalone or maybe it becomes part of VDSM, either way the path forward should start with adding the project into oVirt.

Aic > > On a related point, there are good examples that argue for > consolidating function inside a single daemon, and there are good > counterexamples. It's not always true that every node policy > function should be integrated within a single daemon. It probably > makes sense in this case. But again, VDSM doesn't do what MOM does > today, which is another argument in favor of contributing MOM and > letting the community work with it. > > Thanks, > > Mike > > Mike Day > IBM Distinguished Engineer > Chief Virtualization Architect, Open Systems Development > Cell: +1 919 371-8786 | mdday(a)us.ibm.com > http://code.ncultra.org > _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

I think this discussion highlights the need for a bit more structure for adding a sub project. I would like to propose the following: 1) Project posts an overview page, including scope and short term roadmap to ovirt.org wiki. That page is then copied as a mail to board@ovirt 2) There is an official comment period that lasts for a fixed period of time (2-4 weeks?). This time period allows for questions to be enhanced, and the wiki to be improved. 3) At any point in this time period, the proposal can be officially withdrawn. 4) At the end of the period, there is a final call for comments before a vote takes place.

On 09/27/2011 10:27 AM, Anthony Liguori wrote: > I think this discussion highlights the need for a bit more structure > for adding a sub project. I would like to propose the following: > > 1) Project posts an overview page, including scope and short term > roadmap to ovirt.org wiki. That page is then copied as a mail to > board@ovirt > > 2) There is an official comment period that lasts for a fixed period > of time (2-4 weeks?). This time period allows for questions to be > enhanced, and the wiki to be improved. > > 3) At any point in this time period, the proposal can be officially > withdrawn. > > 4) At the end of the period, there is a final call for comments before > a vote takes place. I like. Don't think we need 2-4 weeks. maybe say discuss till basic agreement is reached. could be 24 hours, could be weeks.

Let's also add the basic questions for the new project to answer, and turn this into an additional page on the site.

Carl. _______________________________________________ Board mailing list Board(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/board

The other thing I was thinking is that could have Acked-by's and wait to get N Acked-by's before going to a vote. The reason for having a time period was to give people time to actually look at code and read proposals instead of just immediately jumping into a heated debate. I think forcing a little think time is a good thing.

> > Let's also add the basic questions for the new project to answer, and > turn this into an additional page on the site. Do we have a wiki ETA? I'd be happy to start putting together questions.

wiki is live. ovirt.org/wiki