From jchoate at redhat.com Tue Nov 1 19:14:56 2011 From: jchoate at redhat.com (Jon Choate) Date: Tue, 01 Nov 2011 15:14:56 -0400 Subject: [Engine-devel] project meetings on irc? Message-ID: <4EB04530.3000401@redhat.com> I'd like to suggest holding our project status meetings on IRC. RichFaces does this (http://community.jboss.org/wiki/richfacesteammeetinginformation) and appears to have great success with it. There is a tool (http://wiki.debian.org/MeetBot) that will record, summarize and publish the minutes of the meeting automagically. Jon From kwade at redhat.com Tue Nov 1 22:45:05 2011 From: kwade at redhat.com (Karsten Wade) Date: Tue, 1 Nov 2011 15:45:05 -0700 Subject: [Engine-devel] project meetings on irc? In-Reply-To: <4EB04530.3000401@redhat.com> References: <4EB04530.3000401@redhat.com> Message-ID: <20111101224454.GB5880@terpsichore.fairy-talefarm.com> On Tue, Nov 01, 2011 at 03:14:56PM -0400, Jon Choate wrote: > I'd like to suggest holding our project status meetings on IRC. > RichFaces does this > (http://community.jboss.org/wiki/richfacesteammeetinginformation) > and appears to have great success with it. There is a tool > (http://wiki.debian.org/MeetBot) that will record, summarize and > publish the minutes of the meeting automagically. I agree - having open meetings in the easiest to access format is crucial to an open source project: http://bit.ly/StartingTOSW [1] BTW, meetbot is on the short-list for Infrastructure projects, probably the first thing I'll tackle when the workshop is clearly All Good. It should be very easy. - Karsten [1] Full fat URL: https://www.theopensourceway.org/wiki/How_to_loosely_organize_a_community#Get_things_started_immediately_with_the_simplest_and_most_open_communication_methods_available_plus_a_meeting_time -- name: Karsten 'quaid' Wade, Sr. Community Gardener team: Red Hat Community Architecture & Leadership uri: http://communityleadershipteam.org http://TheOpenSourceWay.org gpg: AD0E0C41 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From lyarwood at redhat.com Wed Nov 2 10:34:57 2011 From: lyarwood at redhat.com (Lee Yarwood) Date: Wed, 02 Nov 2011 10:34:57 +0000 Subject: [Engine-devel] project meetings on irc? In-Reply-To: <20111101224454.GB5880@terpsichore.fairy-talefarm.com> References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> Message-ID: <4EB11CD1.5090901@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/11/11 22:45, Karsten Wade wrote: > BTW, meetbot is on the short-list for Infrastructure projects, > probably the first thing I'll tackle when the workshop is clearly All > Good. It should be very easy. Is this list for the engine project only or for all ovirt projects? I'd love to help out if possible, can we publish this list anywhere? Thanks, Lee - -- Lee Yarwood, RHCE Technical Support Engineer Red Hat UK Ltd 200 Fowler Avenue IQ Farnborough, Farnborough, Hants GU14 7JP Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham (USA), Brendan Lane (Ireland), Matt Parson(USA), Charlie Peters (USA) GPG fingerprint : A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOsRzRAAoJELymbjP2ci12oCoIAKfhtmEVZZZB5RYWOgFI+W0V TELXQZ2k53IDq5LWxZVi89aVKNj845jpsgpZpRCjrGmsfl+JuwTTPB6cQFj32tz9 IMCv66XKCxT6VDidlbLsa25x+icHNX8+dTnb2TH8Gu4LAf+9sO7XE74hyMy2rRYY ssKaBM1PDm5FYQ+hbcQiS6ZG1z8zg70n86OuGQB/e7hazlQp09/fPJwJhWRztcdS NheFUFLBb9xkeBKk9UEyMSRhGE1pdon6/NRX9x/4MwmkMk1ZMN5OaI6e6eKQZQ3V D7MSlHdrnYkKOWpQ3dJtC9EJyZ0EzysbjHi/XNps2A2GkRRg3Sjb1JWjz7HbTjU= =Df7i -----END PGP SIGNATURE----- From iheim at redhat.com Wed Nov 2 11:54:42 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 02 Nov 2011 13:54:42 +0200 Subject: [Engine-devel] project meetings on irc? In-Reply-To: <4EB11CD1.5090901@redhat.com> References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> <4EB11CD1.5090901@redhat.com> Message-ID: <4EB12F82.1040009@redhat.com> On 11/02/2011 12:34 PM, Lee Yarwood wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/11/11 22:45, Karsten Wade wrote: >> BTW, meetbot is on the short-list for Infrastructure projects, >> probably the first thing I'll tackle when the workshop is clearly All >> Good. It should be very easy. > > Is this list for the engine project only or for all ovirt projects? currently it covers most aspects of the engine. we'll probably consider splitting out specific topics to their own mailing lists going forward. > > I'd love to help out if possible, can we publish this list anywhere? they are all public: http://www.ovirt.org/project/subprojects/ > > Thanks, > > Lee > - -- > > Lee Yarwood, RHCE > Technical Support Engineer > Red Hat UK Ltd > 200 Fowler Avenue IQ Farnborough, Farnborough, Hants GU14 7JP > > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham (USA), Brendan Lane (Ireland), Matt > Parson(USA), Charlie Peters (USA) > > GPG fingerprint : A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJOsRzRAAoJELymbjP2ci12oCoIAKfhtmEVZZZB5RYWOgFI+W0V > TELXQZ2k53IDq5LWxZVi89aVKNj845jpsgpZpRCjrGmsfl+JuwTTPB6cQFj32tz9 > IMCv66XKCxT6VDidlbLsa25x+icHNX8+dTnb2TH8Gu4LAf+9sO7XE74hyMy2rRYY > ssKaBM1PDm5FYQ+hbcQiS6ZG1z8zg70n86OuGQB/e7hazlQp09/fPJwJhWRztcdS > NheFUFLBb9xkeBKk9UEyMSRhGE1pdon6/NRX9x/4MwmkMk1ZMN5OaI6e6eKQZQ3V > D7MSlHdrnYkKOWpQ3dJtC9EJyZ0EzysbjHi/XNps2A2GkRRg3Sjb1JWjz7HbTjU= > =Df7i > -----END PGP SIGNATURE----- > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From lyarwood at redhat.com Wed Nov 2 12:05:57 2011 From: lyarwood at redhat.com (Lee Yarwood) Date: Wed, 02 Nov 2011 12:05:57 +0000 Subject: [Engine-devel] project meetings on irc? In-Reply-To: <4EB12F82.1040009@redhat.com> References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> <4EB11CD1.5090901@redhat.com> <4EB12F82.1040009@redhat.com> Message-ID: <4EB13225.7090304@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/11/11 11:54, Itamar Heim wrote: > On 11/02/2011 12:34 PM, Lee Yarwood wrote: > On 01/11/11 22:45, Karsten Wade wrote: >>>> BTW, meetbot is on the short-list for Infrastructure projects, >>>> probably the first thing I'll tackle when the workshop is clearly All >>>> Good. It should be very easy. > > Is this list for the engine project only or for all ovirt projects? > >> currently it covers most aspects of the engine. we'll probably consider >> splitting out specific topics to their own mailing lists going forward. > > > I'd love to help out if possible, can we publish this list anywhere? > >> they are all public: >> http://www.ovirt.org/project/subprojects/ Thanks Itamar, I was actually after a list of infrastructure projects such as meetbot etc that I could help out with if possible. Sorry for the confusion with my use of the term 'list' above. Lee > > > Thanks, > > Lee - -- Lee Yarwood, RHCE Technical Support Engineer Red Hat UK Ltd 200 Fowler Avenue IQ Farnborough, Farnborough, Hants GU14 7JP Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham (USA), Brendan Lane (Ireland), Matt Parson(USA), Charlie Peters (USA) GPG fingerprint : A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOsTIlAAoJELymbjP2ci12Q7YIAK6/LPMgK/dteR4cacANzGMx tBzCVu8/RJvi5iFJdQPpWoH2u7MIiD5a5RcX9Z7zh+wTrXeevxlxZ82gRbupof9r bTY4dxT2QSyhgQicePoW7C1wXpauqyNSq9sORztS0ru5a3lDPAR9VMO0Ra7jsB1c mQIpiUrlKdiOXjHrrlMyXh9s7PXqXBRzjiMoUXMlUbefQ5tx9W2pKjUkk768YOyT x70Lj/mSxyo0d82OlBrObhnj5K5jpaHUTIWW33rTKP239CFGV6V2l1B/0af+6TwR OvWUJlx8onUEQciadkZJevYd/l+jak6e4Qp9aEpKXe5fg0BRq3/7lOgz77vf2gQ= =TOLe -----END PGP SIGNATURE----- From apevec at gmail.com Wed Nov 2 15:10:55 2011 From: apevec at gmail.com (Alan Pevec) Date: Wed, 2 Nov 2011 08:10:55 -0700 Subject: [Engine-devel] project meetings on irc? In-Reply-To: <4EB13225.7090304@redhat.com> References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> <4EB11CD1.5090901@redhat.com> <4EB12F82.1040009@redhat.com> <4EB13225.7090304@redhat.com> Message-ID: On Wed, Nov 2, 2011 at 5:05 AM, Lee Yarwood wrote: > I was actually after a list of infrastructure projects such as meetbot > etc that I could help out with if possible. that would be infra list http://lists.ovirt.org/mailman/listinfo/infra All ovirt lists are at http://lists.ovirt.org/mailman/listinfo Alan From jchoate at redhat.com Wed Nov 2 15:16:04 2011 From: jchoate at redhat.com (Jon Choate) Date: Wed, 02 Nov 2011 11:16:04 -0400 Subject: [Engine-devel] project meetings on irc? In-Reply-To: References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> <4EB11CD1.5090901@redhat.com> <4EB12F82.1040009@redhat.com> <4EB13225.7090304@redhat.com> Message-ID: <4EB15EB4.5040109@redhat.com> On 11/02/2011 11:10 AM, Alan Pevec wrote: > On Wed, Nov 2, 2011 at 5:05 AM, Lee Yarwood wrote: >> I was actually after a list of infrastructure projects such as meetbot >> etc that I could help out with if possible. > that would be infra list http://lists.ovirt.org/mailman/listinfo/infra > > All ovirt lists are at http://lists.ovirt.org/mailman/listinfo > > Alan > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel I don't think that is what he is looking for either. I believe he is looking for Karsten's "short-list for Infrastructure projects" and not a mailing list. From kwade at redhat.com Wed Nov 2 15:38:11 2011 From: kwade at redhat.com (Karsten Wade) Date: Wed, 2 Nov 2011 08:38:11 -0700 Subject: [Engine-devel] project meetings on irc? In-Reply-To: References: <4EB04530.3000401@redhat.com> <20111101224454.GB5880@terpsichore.fairy-talefarm.com> <4EB11CD1.5090901@redhat.com> <4EB12F82.1040009@redhat.com> <4EB13225.7090304@redhat.com> Message-ID: <20111102153811.GD2891@terpsichore.fairy-talefarm.com> On Wed, Nov 02, 2011 at 08:10:55AM -0700, Alan Pevec wrote: > On Wed, Nov 2, 2011 at 5:05 AM, Lee Yarwood wrote: > > I was actually after a list of infrastructure projects such as meetbot > > etc that I could help out with if possible. > > that would be infra list http://lists.ovirt.org/mailman/listinfo/infra > > All ovirt lists are at http://lists.ovirt.org/mailman/listinfo We're also doing some organizing here on the wiki: http://ovirt.org/wiki/Infrastructure About meetbot, I planned to use supybot-meetbot from EPEL, and I have some nice, specific instructions from another meetbot admin. What would be a good exercise is to discuss the how-we-want-it-done as a group on infra@, then do it. :) - Karsten -- name: Karsten 'quaid' Wade, Sr. Community Gardener team: Red Hat Community Architecture & Leadership uri: http://communityleadershipteam.org http://TheOpenSourceWay.org gpg: AD0E0C41 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From iheim at redhat.com Sun Nov 6 11:53:52 2011 From: iheim at redhat.com (Itamar Heim) Date: Sun, 06 Nov 2011 13:53:52 +0200 Subject: [Engine-devel] gerrit.ovirt.org now has gitweb Message-ID: <4EB67550.90608@redhat.com> gerrit will show a 'gitweb' link next to the patch set when reviewing a patch. you can use gitweb directly as well: http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git From dfediuck at redhat.com Mon Nov 7 04:21:34 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Mon, 7 Nov 2011 06:21:34 +0200 Subject: [Engine-devel] Beware of VM MAC address Message-ID: <201111070621.34662.dfediuck@redhat.com> Hi all, This mail is intended to warn people from mac address issues, and determine the best way to handle it. When you create a new VM using the engine core, its MAC address is being selected from a range of allowed addresses in the the DB. (MacPoolRanges in vdc_options table). As developers and users, you must verify the range does not conflict in anyway with existing machines (both physical and virtual), or this may lead to networking issues. _So before running a VM, please make sure it will not create issues._ The default range is 00:1A:4A:16:01:51-00:1A:4A:16:01:e6, which belongs to Qumranet and we may move to some unassigned prefix. We'd still need a way to randomize this, to prevent colleagues on the same network from collision. Here's a sample on how this is documented and used in qemu-kvm: http://www.linux-kvm.org/page/Networking#private_virtual_bridge (look for 'Generate a MAC address'). -- /d "This message will self destruct in the future. Or not." From mburns at redhat.com Mon Nov 7 13:18:32 2011 From: mburns at redhat.com (Mike Burns) Date: Mon, 07 Nov 2011 08:18:32 -0500 Subject: [Engine-devel] gerrit.ovirt.org now has gitweb In-Reply-To: <4EB67550.90608@redhat.com> References: <4EB67550.90608@redhat.com> Message-ID: <1320671912.28315.20.camel@beelzebub.mburnsfire.net> On Sun, 2011-11-06 at 13:53 +0200, Itamar Heim wrote: > gerrit will show a 'gitweb' link next to the patch set when reviewing a > patch. > > you can use gitweb directly as well: > http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git > _______________________________________________ > Infra mailing list > Infra at ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra Hi Itamar, Any chance the summary page can be made to work? Currently if you go to gerrit.ovirt.org/gitweb, you get redirected to the gerrit interface rather than the gitweb summary of projects. It looks like just a re-write rule problem from the surface. The summary page the I'm looking for is something similar to http://git.kernel.org/ Thanks Mike From dfediuck at redhat.com Mon Nov 7 09:34:31 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Mon, 7 Nov 2011 11:34:31 +0200 Subject: [Engine-devel] Registration & Bootsrapping session Message-ID: <201111071134.31659.dfediuck@redhat.com> Hi guys, Attached is a presentation I made for Registration & Bootsrapping breakout session. Although eventually it didn't happen, I still think you may find it interesting. -- /d "Air conditioned environment - Do NOT open Windows!" -------------- next part -------------- A non-text attachment was scrubbed... Name: ovirt-bootstrap.pdf Type: application/pdf Size: 231374 bytes Desc: not available URL: From dfediuck at redhat.com Mon Nov 7 09:35:46 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Mon, 7 Nov 2011 11:35:46 +0200 Subject: [Engine-devel] Java based tools session Message-ID: <201111071135.46802.dfediuck@redhat.com> Yet another session I prepared which did not take place. Enjoy the slides. -- /d Why doesn't DOS ever say "EXCELLENT command or filename!" -------------- next part -------------- A non-text attachment was scrubbed... Name: ovirt-engin-tools.pdf Type: application/pdf Size: 189860 bytes Desc: not available URL: From iheim at redhat.com Mon Nov 7 13:58:46 2011 From: iheim at redhat.com (Itamar Heim) Date: Mon, 07 Nov 2011 15:58:46 +0200 Subject: [Engine-devel] gerrit.ovirt.org now has gitweb In-Reply-To: <1320671912.28315.20.camel@beelzebub.mburnsfire.net> References: <4EB67550.90608@redhat.com> <1320671912.28315.20.camel@beelzebub.mburnsfire.net> Message-ID: <4EB7E416.5030502@redhat.com> On 11/07/2011 03:18 PM, Mike Burns wrote: > Currently if you go to > gerrit.ovirt.org/gitweb, you get redirected to the gerrit interface > rather than the gitweb summary of projects. It looks like just a > re-write rule problem from the surface. > > The summary page the I'm looking for is something similar to > http://git.kernel.org/ didn't find on google. posted your question on gerrit mailing list for insights. From iheim at redhat.com Mon Nov 7 19:21:47 2011 From: iheim at redhat.com (Itamar Heim) Date: Mon, 07 Nov 2011 21:21:47 +0200 Subject: [Engine-devel] where are the discussions on the patches? Message-ID: <4EB82FCB.1000100@redhat.com> a friendly reminder that all discussions on the patches are done via http://gerrit.ovirt.org anyone with an OpenID identity can login and post comments on patches all patches are also sent to engine-patches at ovirt.org: http://lists.ovirt.org/mailman/listinfo/engine-patches (the mailing list is "read only" - please comment on patches on gerrit, or for fundamental issues, take the discussion back here in engine-devel. Thanks, Itamar From dfediuck at redhat.com Tue Nov 8 12:09:07 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Tue, 8 Nov 2011 14:09:07 +0200 Subject: [Engine-devel] IBM Brings Message Queuing Telemetry Transport to Eclipse - Developer.com Message-ID: <201111081409.08098.dfediuck@redhat.com> This may be of some interest... http://www.developer.com/open/ibm-brings-message-queuing-telemetry-transport-to-eclipse.html -- /d "Ford," he said, "you're turning into a penguin. Stop it." --Douglas Adams, The Hitchhiker's Guide to the Galaxy From cctrieloff at redhat.com Tue Nov 8 14:16:45 2011 From: cctrieloff at redhat.com (Carl Trieloff) Date: Tue, 08 Nov 2011 09:16:45 -0500 Subject: [Engine-devel] where are the discussions on the patches? In-Reply-To: <4EB82FCB.1000100@redhat.com> References: <4EB82FCB.1000100@redhat.com> Message-ID: <4EB939CD.4050209@redhat.com> As an indication, there have been over 40 messages on the patches list already today. Carl. On 11/07/2011 02:21 PM, Itamar Heim wrote: > a friendly reminder that all discussions on the patches are done via > http://gerrit.ovirt.org > > anyone with an OpenID identity can login and post comments on patches > > all patches are also sent to engine-patches at ovirt.org: > http://lists.ovirt.org/mailman/listinfo/engine-patches > > (the mailing list is "read only" - please comment on patches on > gerrit, or for fundamental issues, take the discussion back here in > engine-devel. > > Thanks, > Itamar > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From ovedo at redhat.com Wed Nov 9 13:29:38 2011 From: ovedo at redhat.com (Oved Ourfalli) Date: Wed, 09 Nov 2011 08:29:38 -0500 (EST) Subject: [Engine-devel] Introducing new Features to oVirt In-Reply-To: <7fa90384-475f-49ce-b92e-b84d137fe19e@zmail02.collab.prod.int.phx2.redhat.com> Message-ID: <9a05fbfa-3aad-442c-9ff5-11e15463123a@zmail02.collab.prod.int.phx2.redhat.com> Hi all. We thought it would be nice if we can have templates for adding new feature to oVirt. Here is an initial proposal for that: - Have a 'features' category - this will include a general info page on the feature: http://www.ovirt.org/wiki/Features/FeatureTemplate - Have a category per release and for each feature it will include: 1. More detailed page on the part of the feature that the owner wants to implement in the release: http://www.ovirt.org/wiki/Features/DetailedFeatureTemplate 2. Technical design page - more low level implementation details. No template for that one yet. Your comments will be appreciated! Oved From smizrahi at redhat.com Thu Nov 10 09:51:06 2011 From: smizrahi at redhat.com (Saggi Mizrahi) Date: Thu, 10 Nov 2011 11:51:06 +0200 Subject: [Engine-devel] New version of git-request-review Message-ID: <4EBB9E8A.1060506@redhat.com> I just finished a version of my git-request-review script. For those who are not yet using it it's meant to make sending patches to gerrit more streamlined. It show you a list of the patches before sending them and also warns you if you are not rebased. To have it working you need to have a configured remote for gerrit already in your git config In order for git to detect it as a new verb you have to have this in your path usage: git request-review eg. git request-review gerrit master For those of you who are using the old version will probably notice this version forgoes special gerrit configuration and can support multiple gerrit servers (just configure more remotes). Fixes, features and comments are always welcome. Happy reviewing. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: git-request-review URL: From danken at redhat.com Thu Nov 10 11:03:40 2011 From: danken at redhat.com (Dan Kenigsberg) Date: Thu, 10 Nov 2011 13:03:40 +0200 Subject: [Engine-devel] New version of git-request-review In-Reply-To: <4EBB9E8A.1060506@redhat.com> References: <4EBB9E8A.1060506@redhat.com> Message-ID: <20111110110338.GA21243@redhat.com> On Thu, Nov 10, 2011 at 11:51:06AM +0200, Saggi Mizrahi wrote: > I just finished a version of my git-request-review script. > > For those who are not yet using it it's meant to make sending > patches to gerrit more streamlined. > It show you a list of the patches before sending them and also warns > you if you are not rebased. > > To have it working you need to have a configured remote for gerrit > already in your git config > In order for git to detect it as a new verb you have to have this in > your path > > usage: git request-review > > eg. git request-review gerrit master > > For those of you who are using the old version will probably notice > this version forgoes special gerrit configuration and can support > multiple gerrit servers (just configure more remotes). > > Fixes, features and comments are always welcome. > > Happy reviewing. > > > #!/bin/bash > # > # Copyright 2011 Saggi Mizrahi > # > # Licensed to you under the GNU General Public License as published by > # the Free Software Foundation; either version 2 of the License, or > # (at your option) any later version. See the files README and > # LICENSE_GPL_v2 which accompany this distribution. Nothing accompanies your script! Better don't mention it. > # > > if [ -z $2 ]; then here, and everywhere, unquoted bash variable is a syntax error waiting to happen. if you go over and fix it, better use names for $1 $2. > echo "usage: git request-review " > exit 1 wouldn't it be nicer to have a config bit for default gerrit-remote? most people are going to have only one. > fi > Thanks Saggi for your script, I use it daily. Dan. From dmacvicar at suse.de Thu Nov 10 13:37:36 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Thu, 10 Nov 2011 14:37:36 +0100 Subject: [Engine-devel] missing artifacts? Message-ID: <4EBBD3A0.3080808@suse.de> Hi guys, At the workshop I remember lot of artifacts were missing in the maven repos. I was told that the settings.xml had a bug and contained an internal repository. I solved it by installing required artifacts by hand. Now I am trying to build it in jenkins, and there are two artifacts missing, any ideas what repos are missing in the settings.xml mentioned in http://ovirt.org/wiki/Building_Ovirt_Engine ? Missing: ---------- 1) org.jboss.security:jbosssx-bare:jar:2.0.4 Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=org.jboss.security -DartifactId=jbosssx-bare -Dversion=2.0.4 -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=org.jboss.security -DartifactId=jbosssx-bare -Dversion=2.0.4 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.ovirt.engine.core:engineencryptutils:jar:3.0.0-0001 2) org.jboss.security:jbosssx-bare:jar:2.0.4 2) sun-jaxb:jaxb-impl:jar:2.2 Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=sun-jaxb -DartifactId=jaxb-impl -Dversion=2.2 -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=sun-jaxb -DartifactId=jaxb-impl -Dversion=2.2 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.ovirt.engine.core:engineencryptutils:jar:3.0.0-0001 2) sun-jaxb:jaxb-impl:jar:2.2 -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From dfediuck at redhat.com Thu Nov 10 13:59:03 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Thu, 10 Nov 2011 15:59:03 +0200 Subject: [Engine-devel] missing artifacts? In-Reply-To: <4EBBD3A0.3080808@suse.de> References: <4EBBD3A0.3080808@suse.de> Message-ID: <201111101559.04225.dfediuck@redhat.com> Hi Duncan, long time, mate ;) In the ovirt wiki link you posted, there's all the relevant entries needed. Specifically these to items can be found in the jboss-public-repository-group repo in the XML: org.jboss.security:jbosssx-bare:jar:2.0.4 https://repository.jboss.org/nexus/content/groups/public/org/jboss/security/jbosssx-bare/2.0.4/ sun-jaxb:jaxb-impl:jar:2.2 https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/ Can you make sure your Jenkins machine can http to this server? Let me know if you need more help. Doron On Thursday 10 November 2011 15:37:36 Duncan Mac-Vicar P. wrote: > > Hi guys, > > At the workshop I remember lot of artifacts were missing in the maven > repos. I was told that the settings.xml had a bug and contained an > internal repository. I solved it by installing required artifacts by hand. > > Now I am trying to build it in jenkins, and there are two artifacts > missing, any ideas what repos are missing in the settings.xml mentioned > in http://ovirt.org/wiki/Building_Ovirt_Engine ? > > Missing: > ---------- > 1) org.jboss.security:jbosssx-bare:jar:2.0.4 > > Try downloading the file manually from the project website. > > Then, install it using the command: > mvn install:install-file -DgroupId=org.jboss.security > -DartifactId=jbosssx-bare -Dversion=2.0.4 -Dpackaging=jar > -Dfile=/path/to/file > > Alternatively, if you host your own repository you can deploy the > file there: > mvn deploy:deploy-file -DgroupId=org.jboss.security > -DartifactId=jbosssx-bare -Dversion=2.0.4 -Dpackaging=jar > -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] > > Path to dependency: > 1) org.ovirt.engine.core:engineencryptutils:jar:3.0.0-0001 > 2) org.jboss.security:jbosssx-bare:jar:2.0.4 > > 2) sun-jaxb:jaxb-impl:jar:2.2 > > Try downloading the file manually from the project website. > > Then, install it using the command: > mvn install:install-file -DgroupId=sun-jaxb > -DartifactId=jaxb-impl -Dversion=2.2 -Dpackaging=jar -Dfile=/path/to/file > > Alternatively, if you host your own repository you can deploy the > file there: > mvn deploy:deploy-file -DgroupId=sun-jaxb -DartifactId=jaxb-impl > -Dversion=2.2 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] > -DrepositoryId=[id] > > Path to dependency: > 1) org.ovirt.engine.core:engineencryptutils:jar:3.0.0-0001 > 2) sun-jaxb:jaxb-impl:jar:2.2 > > -- /d "Do not look into laser with remaining eye." --On a laser pointer user-manual From dmacvicar at suse.de Thu Nov 10 14:13:25 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Thu, 10 Nov 2011 15:13:25 +0100 Subject: [Engine-devel] missing artifacts? In-Reply-To: <201111101559.04225.dfediuck@redhat.com> References: <4EBBD3A0.3080808@suse.de> <201111101559.04225.dfediuck@redhat.com> Message-ID: <4EBBDC05.3020108@suse.de> On 11/10/2011 02:59 PM, Doron Fediuck wrote: > Hi Duncan, > long time, mate ;) Hey :-) > In the ovirt wiki link you posted, there's all the relevant entries > needed. Specifically these to items can be found in the jboss-public-repository-group repo in the XML: > > org.jboss.security:jbosssx-bare:jar:2.0.4 > https://repository.jboss.org/nexus/content/groups/public/org/jboss/security/jbosssx-bare/2.0.4/ > > sun-jaxb:jaxb-impl:jar:2.2 > https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/ > > Can you make sure your Jenkins machine can http to this server? > Let me know if you need more help. Downloading: https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar [WARNING] Unable to get resource 'sun-jaxb:jaxb-impl:jar:2.2' from repository atlassian.public.repo (https://maven.atlassian.com/content/groups/public): Error transferring file: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty I can't get it from the browser or wget either: (typing url) https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar The server has not found anything matching the request URI You can get technical details here. Please continue your visit at our home page. wget: Connecting to maven.atlassian.com (maven.atlassian.com)|67.221.237.4|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2011-11-10 15:12:25 ERROR 404: Not Found. -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From dfediuck at redhat.com Thu Nov 10 15:23:26 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Thu, 10 Nov 2011 17:23:26 +0200 Subject: [Engine-devel] missing artifacts? In-Reply-To: <4EBBDC05.3020108@suse.de> References: <4EBBD3A0.3080808@suse.de> <201111101559.04225.dfediuck@redhat.com> <4EBBDC05.3020108@suse.de> Message-ID: <201111101723.26617.dfediuck@redhat.com> On Thursday 10 November 2011 16:13:25 Duncan Mac-Vicar P. wrote: > On 11/10/2011 02:59 PM, Doron Fediuck wrote: > > Hi Duncan, > > long time, mate ;) > > Hey :-) > > > In the ovirt wiki link you posted, there's all the relevant entries > > needed. Specifically these to items can be found in the jboss-public-repository-group repo in the XML: > > > > org.jboss.security:jbosssx-bare:jar:2.0.4 > > https://repository.jboss.org/nexus/content/groups/public/org/jboss/security/jbosssx-bare/2.0.4/ > > > > sun-jaxb:jaxb-impl:jar:2.2 > > https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/ > > > > Can you make sure your Jenkins machine can http to this server? > > Let me know if you need more help. > > Downloading: > https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > [WARNING] Unable to get resource 'sun-jaxb:jaxb-impl:jar:2.2' from > repository atlassian.public.repo > (https://maven.atlassian.com/content/groups/public): Error transferring > file: java.lang.RuntimeException: Unexpected error: > java.security.InvalidAlgorithmParameterException: the trustAnchors > parameter must be non-empty > > I can't get it from the browser or wget either: > > (typing url) > https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > > The server has not found anything matching the request URI > > You can get technical details here. > Please continue your visit at our home page. > > wget: > Connecting to maven.atlassian.com > (maven.atlassian.com)|67.221.237.4|:443... connected. > HTTP request sent, awaiting response... 404 Not Found > 2011-11-10 15:12:25 ERROR 404: Not Found. > > Duncan, both work for me using wget: wget https://repository.jboss.org/nexus/content/groups/public/org/jboss/security/jbosssx-bare/2.0.4/jbosssx-bare-2.0.4.jar --2011-11-10 16:59:11-- https://repository.jboss.org/nexus/content/groups/public/org/jboss/security/jbosssx-bare/2.0.4/jbosssx-bare-2.0.4.jar Resolving repository.jboss.org... 10.5.105.13 Connecting to repository.jboss.org|10.5.105.13|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 553881 (541K) [application/java-archive] Saving to: `jbosssx-bare-2.0.4.jar' 100%[==============================================================================================================================>] 553,881 68.6K/s in 8.1s 2011-11-10 16:59:21 (66.5 KB/s) - `jbosssx-bare-2.0.4.jar' saved [553881/553881] and- wget https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar --2011-11-10 17:12:15-- https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar Resolving repository.jboss.org... 10.5.105.13 Connecting to repository.jboss.org|10.5.105.13|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 872839 (852K) [application/java-archive] Saving to: `jaxb-impl-2.2.jar' 100%[==============================================================================================================================>] 872,839 53.9K/s in 23s 2011-11-10 17:12:40 (37.6 KB/s) - `jaxb-impl-2.2.jar' saved [872839/872839] Now, checking your output again, I see you're using the following url- https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar While I'm using the wiki's url- https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar Can you please verify your setting.xml file? -- /d "Email returned to sender -- insufficient voltage." -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmacvicar at suse.de Fri Nov 11 06:47:39 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Fri, 11 Nov 2011 07:47:39 +0100 Subject: [Engine-devel] missing artifacts? In-Reply-To: <201111101723.26617.dfediuck@redhat.com> References: <4EBBD3A0.3080808@suse.de> <201111101559.04225.dfediuck@redhat.com> <4EBBDC05.3020108@suse.de> <201111101723.26617.dfediuck@redhat.com> Message-ID: <4EBCC50B.2060506@suse.de> On 11/10/2011 04:23 PM, Doron Fediuck wrote: > 2011-11-10 17:12:40 (37.6 KB/s) - `jaxb-impl-2.2.jar' saved [872839/872839] > > > Now, checking your output again, I see you're using the following url- > https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > While I'm using the wiki's url- > https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > > Can you please verify your setting.xml file? I verified and I had the right servers. Googling for the error I found: http://jenkins.361315.n4.nabble.com/Plugins-Upgrades-java-security-InvalidAlgorithmParameterException-the-trustAnchors-parameter-must-bey-td381495.html And Kohsuke related it to openjdk. I changed the build node jvm to sun and much better. All modules are now green, a couple gray, and one red: userportal, which seems to be related to the stack size: [INFO] at com.google.gwt.user.rebind.ClassSourceFileComposer.print(ClassSourceFileComposer.java:162) [INFO] [ERROR] Stack overflow; to increase the stack size, use the -Xss flag at startup (java -Xss1M ...) But that is something I can be solved following the error message :-) Thanks! -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From dfediuck at redhat.com Fri Nov 11 06:59:44 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Fri, 11 Nov 2011 01:59:44 -0500 (EST) Subject: [Engine-devel] =?utf-8?q?missing_artifacts=3F?= Message-ID: <1930755867.2357403.1320994784830.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Hi Duncan, Thanks for finding it. It would be nice if you add a comment to the wiki, so other users may be aware of it and maybe make it work with openjdk. It seems that the root cause was related to certificates. Sent from my Android phone. Please ignore typos. -----Original Message----- From: Duncan Mac-Vicar P. [dmacvicar at suse.de] Received: Friday, 11 Nov 2011, 8:47 To: engine-devel at ovirt.org Subject: Re: [Engine-devel] missing artifacts? On 11/10/2011 04:23 PM, Doron Fediuck wrote: > 2011-11-10 17:12:40 (37.6 KB/s) - `jaxb-impl-2.2.jar' saved [872839/872839] > > > Now, checking your output again, I see you're using the following url- > https://maven.atlassian.com/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > While I'm using the wiki's url- > https://repository.jboss.org/nexus/content/groups/public/sun-jaxb/jaxb-impl/2.2/jaxb-impl-2.2.jar > > Can you please verify your setting.xml file? I verified and I had the right servers. Googling for the error I found: http://jenkins.361315.n4.nabble.com/Plugins-Upgrades-java-security-InvalidAlgorithmParameterException-the-trustAnchors-parameter-must-bey-td381495.html And Kohsuke related it to openjdk. I changed the build node jvm to sun and much better. All modules are now green, a couple gray, and one red: userportal, which seems to be related to the stack size: [INFO] at com.google.gwt.user.rebind.ClassSourceFileComposer.print(ClassSourceFileComposer.java:162) [INFO] [ERROR] Stack overflow; to increase the stack size, use the -Xss flag at startup (java -Xss1M ...) But that is something I can be solved following the error message :-) Thanks! -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany _______________________________________________ Engine-devel mailing list Engine-devel at ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel Sent from my Android phone. Please ignore typos. From dmacvicar at suse.de Fri Nov 11 10:30:28 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Fri, 11 Nov 2011 11:30:28 +0100 Subject: [Engine-devel] missing artifacts? In-Reply-To: <1930755867.2357403.1320994784830.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> References: <1930755867.2357403.1320994784830.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <4EBCF944.3040101@suse.de> On 11/11/2011 07:59 AM, Doron Fediuck wrote: > Hi Duncan, > Thanks for finding it. > It would be nice if you add a comment to the wiki, so other > users may be aware of it and > maybe make it work with openjdk. > It seems that the root cause was > related to certificates. Yes, it may be related to the fact that the node was a JeOS, therefore lot of packages were missing. I struggled afterward with memory. The node had 2G and it was not enough: types/builds/2011-11-11_08-58-09/archive/org.ovirt.engine.api/restapi-types/3.0.0-0001/restapi-types-3.0.0-0001.pom [INFO] ------------------------------------------------------------------------ [ERROR] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Compilation failure Failure executing javac, but could not parse the error: The system is out of resources. Consult the following stack trace for details. java.lang.OutOfMemoryError: PermGen space Then I tried on my machine, with 3G (the laptop I used at the workshop to build it had 3G too), but I got also allocation errors. [INFO] Compiling 6 permutations [INFO] [ERROR] Unable to start external process [INFO] java.io.IOException: Cannot run program "/usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/bin/java": java.io.IOException: error=12, Cannot allocate memory [INFO] at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) However, I increased then the Jenkins node to 4G and added: -Xms1024M -Xmx1024M -XX:PermSize=256M -XX:MaxPermSize=256M And then I had a successful build. 4G required! omg... I would like to add this to the wiki, but it does not let me create an account. :-) Duncan From dfediuck at redhat.com Fri Nov 11 12:23:01 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Fri, 11 Nov 2011 07:23:01 -0500 (EST) Subject: [Engine-devel] =?utf-8?q?missing_artifacts=3F?= Message-ID: <688809841.2371640.1321014181740.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Sent from my Android phone. Please ignore typos. -----Original Message----- From: Duncan Mac-Vicar P. [dmacvicar at suse.de] Received: Friday, 11 Nov 2011, 12:30 To: engine-devel at ovirt.org Subject: Re: [Engine-devel] missing artifacts? On 11/11/2011 07:59 AM, Doron Fediuck wrote: > Hi Duncan, > Thanks for finding it. > It would be nice if you add a comment to the wiki, so other > users may be aware of it and > maybe make it work with openjdk. > It seems that the root cause was > related to certificates. Yes, it may be related to the fact that the node was a JeOS, therefore lot of packages were missing. I struggled afterward with memory. The node had 2G and it was not enough: types/builds/2011-11-11_08-58-09/archive/org.ovirt.engine.api/restapi-types/3.0.0-0001/restapi-types-3.0.0-0001.pom [INFO] ------------------------------------------------------------------------ [ERROR] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Compilation failure Failure executing javac, but could not parse the error: The system is out of resources. Consult the following stack trace for details. java.lang.OutOfMemoryError: PermGen space Then I tried on my machine, with 3G (the laptop I used at the workshop to build it had 3G too), but I got also allocation errors. [INFO] Compiling 6 permutations [INFO] [ERROR] Unable to start external process [INFO] java.io.IOException: Cannot run program "/usr/lib64/jvm/java-1.6.0-openjdk-1.6.0/jre/bin/java": java.io.IOException: error=12, Cannot allocate memory [INFO] at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) However, I increased then the Jenkins node to 4G and added: -Xms1024M -Xmx1024M -XX:PermSize=256M -XX:MaxPermSize=256M And then I had a successful build. 4G required! omg... I would like to add this to the wiki, but it does not let me create an account. :-) Duncan Hi Duncan, account created. See private mail for the details. As for mvn settings, I saw it built on a vm with 2 gb in the workshop. Maybe dun clean first and then run another mvn for the build. Sent from my Android phone. Please ignore typos. From dmacvicar at suse.de Fri Nov 11 13:44:55 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Fri, 11 Nov 2011 14:44:55 +0100 Subject: [Engine-devel] how to execute deploy directly after build Message-ID: <4EBD26D7.2080008@suse.de> Hi! When I do mvn clean install... is there a way to do the step cd ear mvn install,dep in one step together with the step above? I would like Jenkins to build and deploy to a instance running in the same node, but I can only specify one goal for maven. Regards, -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From lpeer at redhat.com Fri Nov 11 14:01:01 2011 From: lpeer at redhat.com (Livnat Peer) Date: Fri, 11 Nov 2011 16:01:01 +0200 Subject: [Engine-devel] how to execute deploy directly after build In-Reply-To: <4EBD26D7.2080008@suse.de> References: <4EBD26D7.2080008@suse.de> Message-ID: <4EBD2A9D.8000102@redhat.com> On 11/11/2011 03:44 PM, Duncan Mac-Vicar P. wrote: > > Hi! > > When I do mvn clean install... > > is there a way to do the step > > cd ear > mvn install,dep > > in one step together with the step above? > > I would like Jenkins to build and deploy to a instance running in the > same node, but I can only specify one goal for maven. > > Regards, > Hi Duncan, sure, the cd ear is not a required step you can do both compile and deploy from the root dir, try - $OVIRT_HOME> mvn install -Pdep you can also skip tests if you want a quick cycle - $OVIRT_HOME> mvn install -Pdep -DskipTests Thanks, Livnat From sgordon at redhat.com Fri Nov 11 19:09:16 2011 From: sgordon at redhat.com (Steve Gordon) Date: Fri, 11 Nov 2011 14:09:16 -0500 (EST) Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <0edcf02c-713f-4218-b080-c382a2f0c644@zmail15.collab.prod.int.phx2.redhat.com> Message-ID: <840005de-243a-4768-8f93-590e8cd3485a@zmail15.collab.prod.int.phx2.redhat.com> Hi all, I am having some trouble building the engine with `mvn clean install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I eventually get indicates a failure in building WebAdmin. I am doing this in a clean F16 environment with a fresh clone of the repository. The javac version is javac 1.6.0_22. http://pastebin.com/VZvUpmTK Any suggestions on how to get past this and successfully building the WebAdmin stuff? Thanks, Steve From dfediuck at redhat.com Sat Nov 12 17:36:26 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Sat, 12 Nov 2011 12:36:26 -0500 (EST) Subject: [Engine-devel] =?utf-8?q?Trouble_building_ovirt_engine_-_exit_cod?= =?utf-8?q?e_137_from=09javac?= Message-ID: <468670232.2535949.1321119386255.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Sent from my Android phone. Please ignore typos. -----Original Message----- From: Steve Gordon [sgordon at redhat.com] Received: Friday, 11 Nov 2011, 21:09 To: engine-devel at ovirt.org Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac Hi all, I am having some trouble building the engine with `mvn clean install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I eventually get indicates a failure in building WebAdmin. I am doing this in a clean F16 environment with a fresh clone of the repository. The javac version is javac 1.6.0_22. http://pastebin.com/VZvUpmTK Any suggestions on how to get past this and successfully building the WebAdmin stuff? Thanks, Steve Hi Steve, I'm not familiar with such an issue. That said I've seen many other issues related to gwt compiler. Let's assume this is a memory issue, and rerun the same command without the clean profile, as you didn't modify the code so far. Try and let us know how it ends. _______________________________________________ Engine-devel mailing list Engine-devel at ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel Sent from my Android phone. Please ignore typos. From lpeer at redhat.com Sun Nov 13 07:40:47 2011 From: lpeer at redhat.com (Livnat Peer) Date: Sun, 13 Nov 2011 09:40:47 +0200 Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <840005de-243a-4768-8f93-590e8cd3485a@zmail15.collab.prod.int.phx2.redhat.com> References: <840005de-243a-4768-8f93-590e8cd3485a@zmail15.collab.prod.int.phx2.redhat.com> Message-ID: <4EBF747F.5030708@redhat.com> On 11/11/2011 09:09 PM, Steve Gordon wrote: > Hi all, > > I am having some trouble building the engine with `mvn clean install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I eventually get indicates a failure in building WebAdmin. I am doing this in a clean F16 environment with a fresh clone of the repository. The javac version is javac 1.6.0_22. > > http://pastebin.com/VZvUpmTK > > Any suggestions on how to get past this and successfully building the WebAdmin stuff? > > Thanks, > > Steve Hi Steve, can you please send the output of $> mvn --version also can you please try running this without the 'clean' phase, i want to see if it is related to some kind of timeout mvn install -Pgwt-admin,gwt-user -DskipTests=true -X Livnat > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From lhornyak at redhat.com Mon Nov 14 07:50:00 2011 From: lhornyak at redhat.com (Laszlo Hornyak) Date: Mon, 14 Nov 2011 02:50:00 -0500 (EST) Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <4EBF747F.5030708@redhat.com> Message-ID: <5ec00109-ccf1-461a-8837-8dc2e0786c14@zmail01.collab.prod.int.phx2.redhat.com> Hi, It could be an error either in java or the libraries it is building on, very environment dependent. Did this javac command leave an crash report behind? Something like hs_err_pidNNNN.log ----- Original Message ----- > From: "Livnat Peer" > To: "Steve Gordon" > Cc: engine-devel at ovirt.org > Sent: Sunday, November 13, 2011 8:40:47 AM > Subject: Re: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac > > On 11/11/2011 09:09 PM, Steve Gordon wrote: > > Hi all, > > > > I am having some trouble building the engine with `mvn clean > > install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I > > eventually get indicates a failure in building WebAdmin. I am > > doing this in a clean F16 environment with a fresh clone of the > > repository. The javac version is javac 1.6.0_22. > > > > http://pastebin.com/VZvUpmTK > > > > Any suggestions on how to get past this and successfully building > > the WebAdmin stuff? > > > > Thanks, > > > > Steve > > > Hi Steve, > > can you please send the output of > > $> mvn --version > > also can you please try running this without the 'clean' phase, i > want > to see if it is related to some kind of timeout > > mvn install -Pgwt-admin,gwt-user -DskipTests=true -X > > > Livnat > > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From jchoate at redhat.com Mon Nov 14 12:09:30 2011 From: jchoate at redhat.com (Jon Choate) Date: Mon, 14 Nov 2011 07:09:30 -0500 Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <5ec00109-ccf1-461a-8837-8dc2e0786c14@zmail01.collab.prod.int.phx2.redhat.com> References: <5ec00109-ccf1-461a-8837-8dc2e0786c14@zmail01.collab.prod.int.phx2.redhat.com> Message-ID: <4EC104FA.3030704@redhat.com> On 11/14/2011 02:50 AM, Laszlo Hornyak wrote: > Hi, > > It could be an error either in java or the libraries it is building on, very environment dependent. Did this javac command leave an crash report behind? Something like hs_err_pidNNNN.log > > > ----- Original Message ----- >> From: "Livnat Peer" >> To: "Steve Gordon" >> Cc: engine-devel at ovirt.org >> Sent: Sunday, November 13, 2011 8:40:47 AM >> Subject: Re: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac >> >> On 11/11/2011 09:09 PM, Steve Gordon wrote: >>> Hi all, >>> >>> I am having some trouble building the engine with `mvn clean >>> install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I >>> eventually get indicates a failure in building WebAdmin. I am >>> doing this in a clean F16 environment with a fresh clone of the >>> repository. The javac version is javac 1.6.0_22. >>> >>> http://pastebin.com/VZvUpmTK >>> >>> Any suggestions on how to get past this and successfully building >>> the WebAdmin stuff? >>> >>> Thanks, >>> >>> Steve >> >> Hi Steve, >> >> can you please send the output of >> >> $> mvn --version >> >> also can you please try running this without the 'clean' phase, i >> want >> to see if it is related to some kind of timeout >> >> mvn install -Pgwt-admin,gwt-user -DskipTests=true -X >> >> >> Livnat >> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel I have seen a few people on irc that have encountered this. I believe one of them was sgtpepper. I suggest you ask there. From sgordon at redhat.com Mon Nov 14 16:11:45 2011 From: sgordon at redhat.com (Steve Gordon) Date: Mon, 14 Nov 2011 11:11:45 -0500 (EST) Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <4EC104FA.3030704@redhat.com> Message-ID: > I have seen a few people on irc that have encountered this. I > believe > one of them was sgtpepper. I suggest you ask there. Hi, I'm already reasonably active on the IRC channel and am aware that sgtpepper was having the same issue, but last I heard he was still trying to get around it which is why I asked here. Thanks, Steve From sgordon at redhat.com Mon Nov 14 17:33:11 2011 From: sgordon at redhat.com (Steve Gordon) Date: Mon, 14 Nov 2011 12:33:11 -0500 (EST) Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <4EBF747F.5030708@redhat.com> Message-ID: ----- Original Message ----- > From: "Livnat Peer" > To: "Steve Gordon" > Cc: engine-devel at ovirt.org > Sent: Sunday, November 13, 2011 2:40:47 AM > Subject: Re: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac > > On 11/11/2011 09:09 PM, Steve Gordon wrote: > > Hi all, > > > > I am having some trouble building the engine with `mvn clean > > install -Pgwt-admin,gwt-user -DskipTests=true -X`, the output I > > eventually get indicates a failure in building WebAdmin. I am > > doing this in a clean F16 environment with a fresh clone of the > > repository. The javac version is javac 1.6.0_22. > > > > http://pastebin.com/VZvUpmTK > > > > Any suggestions on how to get past this and successfully building > > the WebAdmin stuff? > > > > Thanks, > > > > Steve > > > Hi Steve, > > can you please send the output of > > $> mvn --version /usr/lib/jvm/java Apache Maven 2.2.1 (rNON-CANONICAL_2011-10-11_11-56_mockbuild; 2011-10-11 07:56:30-0400) Java version: 1.6.0_22 Java home: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux" version: "3.1.0-7.fc16.x86_64" arch: "amd64" Family: "unix" > also can you please try running this without the 'clean' phase, i > want > to see if it is related to some kind of timeout > > mvn install -Pgwt-admin,gwt-user -DskipTests=true -X I'm running this again now, only have access to the laptop today so will probably be a while until I can report back :). > Livnat > > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > From sgordon at redhat.com Mon Nov 14 22:47:57 2011 From: sgordon at redhat.com (Steve Gordon) Date: Mon, 14 Nov 2011 17:47:57 -0500 (EST) Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: Message-ID: <7973b858-945f-4fbe-87d4-ede94e3c65f6@zmail15.collab.prod.int.phx2.redhat.com> > > $> mvn --version > > /usr/lib/jvm/java > Apache Maven 2.2.1 (rNON-CANONICAL_2011-10-11_11-56_mockbuild; > 2011-10-11 07:56:30-0400) > Java version: 1.6.0_22 > Java home: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux" version: "3.1.0-7.fc16.x86_64" arch: "amd64" Family: > "unix" > > > also can you please try running this without the 'clean' phase, i > > want > > to see if it is related to some kind of timeout > > > > mvn install -Pgwt-admin,gwt-user -DskipTests=true -X > > I'm running this again now, only have access to the laptop today so > will probably be a while until I can report back :). > > > Livnat This produced the same result (error). Based on Doron's suggestion earlier in the thread I decided to throw more memory at the problem. I then received a java.lang.OutOfMemoryError: PermGen space error, setting MAVEN_OPTS="-XX:MaxPermSize=128m" seems to have fixed this and I have now built sucessfully. Thankyou for your suggestions! I have added the perm gen space error tip to http://www.ovirt.org/wiki/Building_oVirt_engine#Build, more generally we probably need to consider some hints/recommendations around what kind of computing power (particularly memory) is required to successfully compile the project. -Steve From mkolesni at redhat.com Tue Nov 15 06:17:27 2011 From: mkolesni at redhat.com (Mike Kolesnik) Date: Tue, 15 Nov 2011 01:17:27 -0500 (EST) Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <4EC15ED6.5070009@redhat.com> Message-ID: Hi, I would like to introduce a change to two tables in oVirt-engine-core. The VM and VM-template share most of their configuration. I am looking to unify the two tables into one table. Currently, if such a change would be made in the DB layer, it would be abstracted by the DAL so no code beyond that layer should be affected, and code changes to that layer should be minimal (since we use views and stored procedures to abstract the DB structure anyway). The major change would be at the DB layer, which will mostly require data migration and changing of the STPs. If anyone has PostgresSQL knowledge and would like to pitch in and help me push this change it would be great. I have more technical details if anyone is interested. Attached is the diff between both tables (vm_static and vm_templates). Regards, Mike -------------- next part -------------- A non-text attachment was scrubbed... Name: vm.template.tables.diff Type: text/x-patch Size: 1499 bytes Desc: not available URL: From lpeer at redhat.com Tue Nov 15 07:09:50 2011 From: lpeer at redhat.com (Livnat Peer) Date: Tue, 15 Nov 2011 09:09:50 +0200 Subject: [Engine-devel] Trouble building ovirt engine - exit code 137 from javac In-Reply-To: <7973b858-945f-4fbe-87d4-ede94e3c65f6@zmail15.collab.prod.int.phx2.redhat.com> References: <7973b858-945f-4fbe-87d4-ede94e3c65f6@zmail15.collab.prod.int.phx2.redhat.com> Message-ID: <4EC2103E.5020207@redhat.com> On 11/15/2011 12:47 AM, Steve Gordon wrote: > >>> $> mvn --version >> >> /usr/lib/jvm/java >> Apache Maven 2.2.1 (rNON-CANONICAL_2011-10-11_11-56_mockbuild; >> 2011-10-11 07:56:30-0400) >> Java version: 1.6.0_22 >> Java home: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre >> Default locale: en_US, platform encoding: UTF-8 >> OS name: "linux" version: "3.1.0-7.fc16.x86_64" arch: "amd64" Family: >> "unix" >> >>> also can you please try running this without the 'clean' phase, i >>> want >>> to see if it is related to some kind of timeout >>> >>> mvn install -Pgwt-admin,gwt-user -DskipTests=true -X >> >> I'm running this again now, only have access to the laptop today so >> will probably be a while until I can report back :). >> >>> Livnat > > This produced the same result (error). Based on Doron's suggestion earlier in the thread I decided to throw more memory at the problem. I then received a java.lang.OutOfMemoryError: PermGen space error, setting MAVEN_OPTS="-XX:MaxPermSize=128m" seems to have fixed this and I have now built sucessfully. Thankyou for your suggestions! > > I have added the perm gen space error tip to http://www.ovirt.org/wiki/Building_oVirt_engine#Build, more generally we probably need to consider some hints/recommendations around what kind of computing power (particularly memory) is required to successfully compile the project. > > -Steve Hi Steve, I Googled around it a little but could not find anything obvious. what i did found is this - http://www.velocityreviews.com/forums/t136593-understanding-error-java-returned-137-a.html "If you use Unix/Linux: Exit-code above 128 means that the process died because of a received signal (exitCode = 128 + signalNumber). ==> In your case it was signal 9 (= SIGKILL)." Not sure it can give you a lead but the next step i would try is compiling without the UI - $OVIRT_HOME> mvn install -DskipTests=true Livnat From iheim at redhat.com Tue Nov 15 08:00:30 2011 From: iheim at redhat.com (Itamar Heim) Date: Tue, 15 Nov 2011 10:00:30 +0200 Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: References: Message-ID: <4EC21C1E.1070005@redhat.com> On 11/15/2011 08:17 AM, Mike Kolesnik wrote: > Hi, > > I would like to introduce a change to two tables in oVirt-engine-core. > > The VM and VM-template share most of their configuration. > I am looking to unify the two tables into one table. > > Currently, if such a change would be made in the DB layer, it would be > abstracted by the DAL so no code beyond that layer should be affected, > and code changes to that layer should be minimal (since we use views and > stored procedures to abstract the DB structure anyway). > The major change would be at the DB layer, which will mostly require > data migration and changing of the STPs. > > If anyone has PostgresSQL knowledge and would like to pitch in and help > me push this change it would be great. > I have more technical details if anyone is interested. > Attached is the diff between both tables (vm_static and vm_templates). > > Regards, > Mike in general, i think this is beneficial. a general concern with merging tables would be table locking, but i am not sure if relevant here. something to think about - how will the model deal with things which will be different between the two? for example, let's say tomorrow we want to add a new feature around templates versions/generations. - user creates a template for templateX - user creates VMs from tempalteX - security and other patches come out, so user wants to update the template so new VMs created from it will have them - user wants to update the template and re-seal it (template is obviously read only, so user needs to create another template). - today user would have to create templateXv1, v2, v3, etc. - would make much more sense if these would all appear as the same template with a version field, rather than disparate templates. going back to the table unification question - maybe it is not a problem, and the table will have a column with the version field which will be relevant only for templates? i guess the question is how many unrelated fields will we have, and will we keep them in same table or separate them out. looking at the fields different right now, i think a single table would be fine. in the future splitting entity specific fields could be revisited. From mkolesni at redhat.com Tue Nov 15 08:15:13 2011 From: mkolesni at redhat.com (Mike Kolesnik) Date: Tue, 15 Nov 2011 03:15:13 -0500 (EST) Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <4EC21C1E.1070005@redhat.com> Message-ID: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > On 11/15/2011 08:17 AM, Mike Kolesnik wrote: > > Hi, > > > > I would like to introduce a change to two tables in > > oVirt-engine-core. > > > > The VM and VM-template share most of their configuration. > > I am looking to unify the two tables into one table. > > > > Currently, if such a change would be made in the DB layer, it would > > be > > abstracted by the DAL so no code beyond that layer should be > > affected, > > and code changes to that layer should be minimal (since we use > > views and > > stored procedures to abstract the DB structure anyway). > > The major change would be at the DB layer, which will mostly > > require > > data migration and changing of the STPs. > > > > If anyone has PostgresSQL knowledge and would like to pitch in and > > help > > me push this change it would be great. > > I have more technical details if anyone is interested. > > Attached is the diff between both tables (vm_static and > > vm_templates). > > > > Regards, > > Mike > > in general, i think this is beneficial. > a general concern with merging tables would be table locking, but i > am > not sure if relevant here. > > something to think about - how will the model deal with things which > will be different between the two? > > for example, let's say tomorrow we want to add a new feature around > templates versions/generations. > - user creates a template for templateX > - user creates VMs from tempalteX > - security and other patches come out, so user wants to update the > template so new VMs created from it will have them > - user wants to update the template and re-seal it (template is > obviously read only, so user needs to create another template). > - today user would have to create templateXv1, v2, v3, etc. > - would make much more sense if these would all appear as the same > template with a version field, rather than disparate templates. > If the templates in presentaion should be grouped by something other than their ID (ie by name) then it can be done at the logic/presentation layer. In the DB I believe that we will still have a new record for each version, and with different UUID so it doesn't affect the table design. > going back to the table unification question - maybe it is not a > problem, and the table will have a column with the version field > which > will be relevant only for templates? > i guess the question is how many unrelated fields will we have, and > will > we keep them in same table or separate them out.\ In this method, "single table inheritance", the fields which are not in the base type are still kept in the same table. This way you gain simplicity and order in the DB, while you give up constraints which need to be kept at the logic level. It's a tradeoff which I think would be good in this case, since the amount of different fields is small. The differnet types simply map to certain fields that they need, much like a view on the table. > > looking at the fields different right now, i think a single table > would > be fine. in the future splitting entity specific fields could be > revisited. > Of course this whole thing can be undone without much work if somewhere along the road we deicde that it wasn't a good idea. Regards, Mike From yzaslavs at redhat.com Tue Nov 15 08:26:57 2011 From: yzaslavs at redhat.com (Yair Zaslavsky) Date: Tue, 15 Nov 2011 10:26:57 +0200 Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> References: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> Message-ID: <4EC22251.9090501@redhat.com> On 11/15/2011 10:15 AM, Mike Kolesnik wrote: > > > ----- Original Message ----- >> On 11/15/2011 08:17 AM, Mike Kolesnik wrote: >>> Hi, >>> >>> I would like to introduce a change to two tables in >>> oVirt-engine-core. >>> >>> The VM and VM-template share most of their configuration. >>> I am looking to unify the two tables into one table. >>> >>> Currently, if such a change would be made in the DB layer, it would >>> be >>> abstracted by the DAL so no code beyond that layer should be >>> affected, >>> and code changes to that layer should be minimal (since we use >>> views and >>> stored procedures to abstract the DB structure anyway). >>> The major change would be at the DB layer, which will mostly >>> require >>> data migration and changing of the STPs. >>> >>> If anyone has PostgresSQL knowledge and would like to pitch in and >>> help >>> me push this change it would be great. >>> I have more technical details if anyone is interested. >>> Attached is the diff between both tables (vm_static and >>> vm_templates). >>> >>> Regards, >>> Mike >> >> in general, i think this is beneficial. >> a general concern with merging tables would be table locking, but i >> am >> not sure if relevant here. >> >> something to think about - how will the model deal with things which >> will be different between the two? >> >> for example, let's say tomorrow we want to add a new feature around >> templates versions/generations. >> - user creates a template for templateX >> - user creates VMs from tempalteX >> - security and other patches come out, so user wants to update the >> template so new VMs created from it will have them >> - user wants to update the template and re-seal it (template is >> obviously read only, so user needs to create another template). >> - today user would have to create templateXv1, v2, v3, etc. >> - would make much more sense if these would all appear as the same >> template with a version field, rather than disparate templates. >> > > If the templates in presentaion should be grouped by something other than their ID (ie by name) then it can be done at the logic/presentation layer. > In the DB I believe that we will still have a new record for each version, and with different UUID so it doesn't affect the table design. > >> going back to the table unification question - maybe it is not a >> problem, and the table will have a column with the version field >> which >> will be relevant only for templates? >> i guess the question is how many unrelated fields will we have, and >> will >> we keep them in same table or separate them out.\ > > In this method, "single table inheritance", the fields which are not in the base type are still kept in the same table. This way you gain simplicity and order in the DB, while you give up constraints which need to be kept at the logic level. It's a tradeoff which I think would be good in this case, since the amount of different fields is small. > > The differnet types simply map to certain fields that they need, much like a view on the table. Just to add on this, this is one of the "inheritance" strategies that is provided by Hibernate/JPA. Another approach is to have all common fields in a single table, and have other tables contain only the fields that are different for each one of the types (and hold an FK to the "common fields" table). Although this approach can be treated as a hybrid between the two, the need for FK here may hit us with performance. > >> >> looking at the fields different right now, i think a single table >> would >> be fine. in the future splitting entity specific fields could be >> revisited. >> > > Of course this whole thing can be undone without much work if somewhere along the road we deicde that it wasn't a good idea. > > > > Regards, > Mike > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From dmacvicar at suse.de Tue Nov 15 08:26:43 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Tue, 15 Nov 2011 09:26:43 +0100 Subject: [Engine-devel] problem trying to add a host Message-ID: <4EC22243.3020404@suse.de> Hi! I am trying to add a host to my engine... 2011-11-15 08:05:01,282 ERROR [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] (http-0.0.0.0-8080-3) Could not connect to server X.X.X.X: Failed connecting to X.X.X.X using given password! Please verify your password is correct and that the host accepts password-based authentication 2011-11-15 08:05:01,282 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (http-0.0.0.0-8080-3) CanDoAction of action AddVds failed. Reasons:VDS_CANNOT_CONNECT_TO_SERVER,VAR__ACTION__ADD,VAR__TYPE__HOST The thing is, I can connect via ssh from the engine to the host, and I can telnet to the host vdsm from the engine: telnet X.X.X.X 54321 Anything else I should check? -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From ofrenkel at redhat.com Tue Nov 15 08:56:36 2011 From: ofrenkel at redhat.com (Omer Frenkel) Date: Tue, 15 Nov 2011 03:56:36 -0500 (EST) Subject: [Engine-devel] problem trying to add a host In-Reply-To: <4EC22243.3020404@suse.de> Message-ID: <254c9a80-a4b7-45c5-b734-3af65484fe80@zmail07.collab.prod.int.phx2.redhat.com> hi Duncan, ----- Original Message ----- > From: "Duncan Mac-Vicar P." > To: engine-devel at ovirt.org > Sent: Tuesday, November 15, 2011 10:26:43 AM > Subject: [Engine-devel] problem trying to add a host > > > Hi! > > I am trying to add a host to my engine... > > 2011-11-15 08:05:01,282 ERROR > [org.ovirt.engine.core.utils.hostinstall.MinaInstallWrapper] > (http-0.0.0.0-8080-3) Could not connect to server X.X.X.X: Failed > connecting to X.X.X.X using given password! Please verify your > password > is correct and that the host accepts password-based authentication did you verify you give correct root password? :) > 2011-11-15 08:05:01,282 WARN > [org.ovirt.engine.core.bll.AddVdsCommand] > (http-0.0.0.0-8080-3) CanDoAction of action AddVds failed. > Reasons:VDS_CANNOT_CONNECT_TO_SERVER,VAR__ACTION__ADD,VAR__TYPE__HOST > > The thing is, I can connect via ssh from the engine to the host, and > I > can telnet to the host vdsm from the engine: telnet X.X.X.X 54321 > > Anything else I should check? > > -- > Duncan Mac-Vicar P. - http://www.suse.com/ > > SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix > Imend?rffer, HRB 16746 (AG N?rnberg) > Maxfeldstra?e 5, 90409 N?rnberg, Germany > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From dmacvicar at suse.de Tue Nov 15 09:40:47 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Tue, 15 Nov 2011 10:40:47 +0100 Subject: [Engine-devel] problem trying to add a host In-Reply-To: <254c9a80-a4b7-45c5-b734-3af65484fe80@zmail07.collab.prod.int.phx2.redhat.com> References: <254c9a80-a4b7-45c5-b734-3af65484fe80@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <4EC2339F.1060406@suse.de> On 11/15/2011 09:56 AM, Omer Frenkel wrote: > did you verify you give correct root password? :) Sure, 5 times, and by two different people. Is the engine at this stage trying a ssh connection or is it a xml-rpc connection to vdsm which does then a pam auth? -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From ofrenkel at redhat.com Tue Nov 15 10:24:41 2011 From: ofrenkel at redhat.com (Omer Frenkel) Date: Tue, 15 Nov 2011 05:24:41 -0500 (EST) Subject: [Engine-devel] problem trying to add a host In-Reply-To: <4EC2339F.1060406@suse.de> Message-ID: <4e0b7148-2f80-464e-b9cb-113081b2d019@zmail07.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > From: "Duncan Mac-Vicar P." > To: engine-devel at ovirt.org > Sent: Tuesday, November 15, 2011 11:40:47 AM > Subject: Re: [Engine-devel] problem trying to add a host > > On 11/15/2011 09:56 AM, Omer Frenkel wrote: > > did you verify you give correct root password? :) > > Sure, 5 times, and by two different people. > > Is the engine at this stage trying a ssh connection or is it a > xml-rpc > connection to vdsm which does then a pam auth? ssh is done with 'root' user here. did you manage to add other hosts? if still doesn't work, you can change the log level of org.ovirt.engine.core.utils.hostinstall to DEBUG (if you need help ping me on irc) > > -- > Duncan Mac-Vicar P. - http://www.suse.com/ > > SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix > Imend?rffer, HRB 16746 (AG N?rnberg) > Maxfeldstra?e 5, 90409 N?rnberg, Germany > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From dfediuck at redhat.com Tue Nov 15 10:37:25 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Tue, 15 Nov 2011 12:37:25 +0200 Subject: [Engine-devel] problem trying to add a host In-Reply-To: <4EC2339F.1060406@suse.de> References: <254c9a80-a4b7-45c5-b734-3af65484fe80@zmail07.collab.prod.int.phx2.redhat.com> <4EC2339F.1060406@suse.de> Message-ID: <201111151237.26319.dfediuck@redhat.com> On Tuesday 15 November 2011 11:40:47 Duncan Mac-Vicar P. wrote: > On 11/15/2011 09:56 AM, Omer Frenkel wrote: > > did you verify you give correct root password? :) > > Sure, 5 times, and by two different people. > > Is the engine at this stage trying a ssh connection or is it a xml-rpc > connection to vdsm which does then a pam auth? > > Hi Duncan, The engine is trying to initiate a simple SSH connection using the credentials you gave it (in this case password). There are 2 main issues if Engine fails to authenticate: 1. Allow root login: you should allow it is it's blocked in sshd_config (PermitRootLogin yes) 2. Authentication type; there's a difference between password auth' methods the ssh /server/ supports. For password based auth, these are mainly Keyboard-interactive and password types (defined in sshd_config file). Please ensure you have (PasswordAuthentication yes) as well as (UsePAM yes). Keyboard-interactive type is unsupported currently. -- /d "Willyoupleasehelpmefixmykeyboard?Thespacebarisbroken!" From emesika at redhat.com Tue Nov 15 11:08:10 2011 From: emesika at redhat.com (Eli Mesika) Date: Tue, 15 Nov 2011 06:08:10 -0500 (EST) Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <4EC22251.9090501@redhat.com> Message-ID: ----- Original Message ----- > From: "Yair Zaslavsky" > To: engine-devel at ovirt.org > Sent: Tuesday, November 15, 2011 10:26:57 AM > Subject: Re: [Engine-devel] Introduce a change to oVirt-engine-core DB > > On 11/15/2011 10:15 AM, Mike Kolesnik wrote: > > > > > > ----- Original Message ----- > >> On 11/15/2011 08:17 AM, Mike Kolesnik wrote: > >>> Hi, > >>> > >>> I would like to introduce a change to two tables in > >>> oVirt-engine-core. > >>> > >>> The VM and VM-template share most of their configuration. > >>> I am looking to unify the two tables into one table. > >>> > >>> Currently, if such a change would be made in the DB layer, it > >>> would > >>> be > >>> abstracted by the DAL so no code beyond that layer should be > >>> affected, > >>> and code changes to that layer should be minimal (since we use > >>> views and > >>> stored procedures to abstract the DB structure anyway). > >>> The major change would be at the DB layer, which will mostly > >>> require > >>> data migration and changing of the STPs. > >>> > >>> If anyone has PostgresSQL knowledge and would like to pitch in > >>> and > >>> help > >>> me push this change it would be great. > >>> I have more technical details if anyone is interested. > >>> Attached is the diff between both tables (vm_static and > >>> vm_templates). > >>> > >>> Regards, > >>> Mike > >> > >> in general, i think this is beneficial. > >> a general concern with merging tables would be table locking, but > >> i > >> am > >> not sure if relevant here. > >> > >> something to think about - how will the model deal with things > >> which > >> will be different between the two? > >> > >> for example, let's say tomorrow we want to add a new feature > >> around > >> templates versions/generations. > >> - user creates a template for templateX > >> - user creates VMs from tempalteX > >> - security and other patches come out, so user wants to update the > >> template so new VMs created from it will have them > >> - user wants to update the template and re-seal it (template is > >> obviously read only, so user needs to create another template). > >> - today user would have to create templateXv1, v2, v3, etc. > >> - would make much more sense if these would all appear as the same > >> template with a version field, rather than disparate templates. > >> > > > > If the templates in presentaion should be grouped by something > > other than their ID (ie by name) then it can be done at the > > logic/presentation layer. > > In the DB I believe that we will still have a new record for each > > version, and with different UUID so it doesn't affect the table > > design. > > > >> going back to the table unification question - maybe it is not a > >> problem, and the table will have a column with the version field > >> which > >> will be relevant only for templates? > >> i guess the question is how many unrelated fields will we have, > >> and > >> will > >> we keep them in same table or separate them out.\ > > > > In this method, "single table inheritance", the fields which are > > not in the base type are still kept in the same table. This way > > you gain simplicity and order in the DB, while you give up > > constraints which need to be kept at the logic level. It's a > > tradeoff which I think would be good in this case, since the > > amount of different fields is small. > > > > The differnet types simply map to certain fields that they need, > > much like a view on the table. > > Just to add on this, this is one of the "inheritance" strategies that > is > provided by Hibernate/JPA. > Another approach is to have all common fields in a single table, and > have other tables contain only the fields that are different for each > one of the types (and hold an FK to the "common fields" table). > Although this approach can be treated as a hybrid between the two, > the > need for FK here may hit us with performance. I have talked with Mike about this whole issue. It seems that we can handle both in one table. We have talked on using a flag to distinguish between both. In case that we have any performance issue, this flag could be used apply partitioning on the table. If diff grows we can use 1) inheritance (several options as above) 2) Handling in the same table (NULL when not relevant) and encapsulate with a view having only relevant columns. 3) having a diff table for each and having a view that joins the information if diff is reasonable 2) is the cheapest/simple solution. if diff grows 1) should be considered I would not use 3) in any case > > > > >> > >> looking at the fields different right now, i think a single table > >> would > >> be fine. in the future splitting entity specific fields could be > >> revisited. > >> > > > > Of course this whole thing can be undone without much work if > > somewhere along the road we deicde that it wasn't a good idea. > > > > > > > > Regards, > > Mike > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From dmacvicar at suse.de Tue Nov 15 11:37:15 2011 From: dmacvicar at suse.de (Duncan Mac-Vicar P.) Date: Tue, 15 Nov 2011 12:37:15 +0100 Subject: [Engine-devel] problem trying to add a host In-Reply-To: <201111151237.26319.dfediuck@redhat.com> References: <254c9a80-a4b7-45c5-b734-3af65484fe80@zmail07.collab.prod.int.phx2.redhat.com> <4EC2339F.1060406@suse.de> <201111151237.26319.dfediuck@redhat.com> Message-ID: <4EC24EEB.8050700@suse.de> On 11/15/2011 11:37 AM, Doron Fediuck wrote: > and password types (defined in sshd_config file). Please ensure you have (PasswordAuthentication yes) > as well as (UsePAM yes). Keyboard-interactive type is unsupported currently. > Thanks Doron. This fixed the issue and the host is now added!. Now I have two AIs (!) Power Management is not configured for this Host. Enable Power Management (not a problem after I figure out all the options of that page) (!) NETWORK_UNREACHABLE Is this the connection from vdsm to the engine? Does it need a port open in the opposite direction? -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany From dfediuck at redhat.com Tue Nov 15 11:45:04 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Tue, 15 Nov 2011 06:45:04 -0500 (EST) Subject: [Engine-devel] =?utf-8?q?problem_trying_to_add_a_host?= Message-ID: <2036544639.2822245.1321357504278.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Sent from my Android phone. Please ignore typos. -----Original Message----- From: Duncan Mac-Vicar P. [dmacvicar at suse.de] Received: Tuesday, 15 Nov 2011, 13:37 To: engine-devel at ovirt.org CC: Jim Fehlig [JFEHLIG at suse.com] Subject: Re: [Engine-devel] problem trying to add a host On 11/15/2011 11:37 AM, Doron Fediuck wrote: > and password types (defined in sshd_config file). Please ensure you have (PasswordAuthentication yes) > as well as (UsePAM yes). Keyboard-interactive type is unsupported currently. > Thanks Doron. This fixed the issue and the host is now added!. Now I have two AIs (!) Power Management is not configured for this Host. Enable Power Management (not a problem after I figure out all the options of that page) (!) NETWORK_UNREACHABLE Is this the connection from vdsm to the engine? Does it need a port open in the opposite direction? You need to create a bridge called engine. Try and let us know. -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg) Maxfeldstra?e 5, 90409 N?rnberg, Germany _______________________________________________ Engine-devel mailing list Engine-devel at ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel Sent from my Android phone. Please ignore typos. From iheim at redhat.com Tue Nov 15 12:20:47 2011 From: iheim at redhat.com (Itamar Heim) Date: Tue, 15 Nov 2011 14:20:47 +0200 Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> References: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> Message-ID: <4EC2591F.9060502@redhat.com> On 11/15/2011 10:15 AM, Mike Kolesnik wrote: > > In this method, "single table inheritance", the fields which are not in the base type are still kept in the same table. This way you gain simplicity and order in the DB, while you give up constraints which need to be kept at the logic level. It's a tradeoff which I think would be good in this case, since the amount of different fields is small. > > The differnet types simply map to certain fields that they need, much like a view on the table. > >> >> looking at the fields different right now, i think a single table >> would >> be fine. in the future splitting entity specific fields could be >> revisited. >> > > Of course this whole thing can be undone without much work if somewhere along the road we deicde that it wasn't a good idea. doesn't have to be undone. you could also just spin off the columns which aren't shared by the two entities. anyway - i think we are agreeing From jchoate at redhat.com Tue Nov 15 14:15:31 2011 From: jchoate at redhat.com (Jon Choate) Date: Tue, 15 Nov 2011 09:15:31 -0500 Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <4EC2591F.9060502@redhat.com> References: <53f636fb-d254-41ca-8146-6939c9e7323e@zmail14.collab.prod.int.phx2.redhat.com> <4EC2591F.9060502@redhat.com> Message-ID: <4EC27403.4000801@redhat.com> On 11/15/2011 07:20 AM, Itamar Heim wrote: > On 11/15/2011 10:15 AM, Mike Kolesnik wrote: >> >> In this method, "single table inheritance", the fields which are not >> in the base type are still kept in the same table. This way you gain >> simplicity and order in the DB, while you give up constraints which >> need to be kept at the logic level. It's a tradeoff which I think >> would be good in this case, since the amount of different fields is >> small. >> >> The differnet types simply map to certain fields that they need, much >> like a view on the table. >> >>> >>> looking at the fields different right now, i think a single table >>> would >>> be fine. in the future splitting entity specific fields could be >>> revisited. >>> >> >> Of course this whole thing can be undone without much work if >> somewhere along the road we deicde that it wasn't a good idea. > > doesn't have to be undone. you could also just spin off the columns > which aren't shared by the two entities. > anyway - i think we are agreeing > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel What problem is this attempting to solve? I understand that it is not aesthetically pleasing to have the two split out but unless this is causing undue complexity in the code (which doesn't seem to be the case due to the abstractions) is causing performance problems or is making further development difficult I'm tempted to say leave it as it is. From mkolesni at redhat.com Tue Nov 15 15:08:01 2011 From: mkolesni at redhat.com (Mike Kolesnik) Date: Tue, 15 Nov 2011 10:08:01 -0500 (EST) Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: <4EC27403.4000801@redhat.com> Message-ID: ----- Original Message ----- > On 11/15/2011 07:20 AM, Itamar Heim wrote: > > On 11/15/2011 10:15 AM, Mike Kolesnik wrote: > >> > >> In this method, "single table inheritance", the fields which are > >> not > >> in the base type are still kept in the same table. This way you > >> gain > >> simplicity and order in the DB, while you give up constraints > >> which > >> need to be kept at the logic level. It's a tradeoff which I think > >> would be good in this case, since the amount of different fields > >> is > >> small. > >> > >> The differnet types simply map to certain fields that they need, > >> much > >> like a view on the table. > >> > >>> > >>> looking at the fields different right now, i think a single table > >>> would > >>> be fine. in the future splitting entity specific fields could be > >>> revisited. > >>> > >> > >> Of course this whole thing can be undone without much work if > >> somewhere along the road we deicde that it wasn't a good idea. > > > > doesn't have to be undone. you could also just spin off the columns > > which aren't shared by the two entities. > > anyway - i think we are agreeing > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > What problem is this attempting to solve? I understand that it is > not > aesthetically pleasing to have the two split out but unless this is > causing undue complexity in the code (which doesn't seem to be the > case > due to the abstractions) is causing performance problems or is making > further development difficult I'm tempted to say leave it as it is. The main benefit I see is keeping it DRY. This will help us manage the DB structure more easily (less tables and mapping tables which all just duplicate each other). I don't think this change is something that heavy that it's not worth doing, and for me keeping it DRY is an advantage, just as you would refactor code to do the same. > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > Regards, Mike From yzaslavs at redhat.com Tue Nov 15 15:37:16 2011 From: yzaslavs at redhat.com (Yair Zaslavsky) Date: Tue, 15 Nov 2011 17:37:16 +0200 Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: References: Message-ID: <4EC2872C.9030405@redhat.com> On 11/15/2011 05:08 PM, Mike Kolesnik wrote: > > ----- Original Message ----- >> On 11/15/2011 07:20 AM, Itamar Heim wrote: >>> On 11/15/2011 10:15 AM, Mike Kolesnik wrote: >>>> >>>> In this method, "single table inheritance", the fields which are >>>> not >>>> in the base type are still kept in the same table. This way you >>>> gain >>>> simplicity and order in the DB, while you give up constraints >>>> which >>>> need to be kept at the logic level. It's a tradeoff which I think >>>> would be good in this case, since the amount of different fields >>>> is >>>> small. >>>> >>>> The differnet types simply map to certain fields that they need, >>>> much >>>> like a view on the table. >>>> >>>>> >>>>> looking at the fields different right now, i think a single table >>>>> would >>>>> be fine. in the future splitting entity specific fields could be >>>>> revisited. >>>>> >>>> >>>> Of course this whole thing can be undone without much work if >>>> somewhere along the road we deicde that it wasn't a good idea. >>> >>> doesn't have to be undone. you could also just spin off the columns >>> which aren't shared by the two entities. >>> anyway - i think we are agreeing >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >> What problem is this attempting to solve? I understand that it is >> not >> aesthetically pleasing to have the two split out but unless this is >> causing undue complexity in the code (which doesn't seem to be the >> case >> due to the abstractions) is causing performance problems or is making >> further development difficult I'm tempted to say leave it as it is. > > The main benefit I see is keeping it DRY. > This will help us manage the DB structure more easily (less tables and mapping tables which all just duplicate each other). > I don't think this change is something that heavy that it's not worth doing, and for me keeping it DRY is an advantage, just as you would refactor code to do the same. +1 On that - I think it will also help new community members to get in our code quicker. > >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > > > Regards, > Mike > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From ofrenkel at redhat.com Tue Nov 15 15:47:36 2011 From: ofrenkel at redhat.com (Omer Frenkel) Date: Tue, 15 Nov 2011 10:47:36 -0500 (EST) Subject: [Engine-devel] Introduce a change to oVirt-engine-core DB In-Reply-To: Message-ID: <27e7e3b3-4e55-4640-84ba-1ec2300c6b3f@zmail07.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > From: "Mike Kolesnik" > To: "Jon Choate" > Cc: engine-devel at ovirt.org > Sent: Tuesday, November 15, 2011 5:08:01 PM > Subject: Re: [Engine-devel] Introduce a change to oVirt-engine-core DB > > > ----- Original Message ----- > > On 11/15/2011 07:20 AM, Itamar Heim wrote: > > > On 11/15/2011 10:15 AM, Mike Kolesnik wrote: > > >> > > >> In this method, "single table inheritance", the fields which are > > >> not > > >> in the base type are still kept in the same table. This way you > > >> gain > > >> simplicity and order in the DB, while you give up constraints > > >> which > > >> need to be kept at the logic level. It's a tradeoff which I > > >> think > > >> would be good in this case, since the amount of different fields > > >> is > > >> small. > > >> > > >> The differnet types simply map to certain fields that they need, > > >> much > > >> like a view on the table. > > >> > > >>> > > >>> looking at the fields different right now, i think a single > > >>> table > > >>> would > > >>> be fine. in the future splitting entity specific fields could > > >>> be > > >>> revisited. > > >>> > > >> > > >> Of course this whole thing can be undone without much work if > > >> somewhere along the road we deicde that it wasn't a good idea. > > > > > > doesn't have to be undone. you could also just spin off the > > > columns > > > which aren't shared by the two entities. > > > anyway - i think we are agreeing > > > _______________________________________________ > > > Engine-devel mailing list > > > Engine-devel at ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > What problem is this attempting to solve? I understand that it is > > not > > aesthetically pleasing to have the two split out but unless this is > > causing undue complexity in the code (which doesn't seem to be the > > case > > due to the abstractions) is causing performance problems or is > > making > > further development difficult I'm tempted to say leave it as it is. > > The main benefit I see is keeping it DRY. > This will help us manage the DB structure more easily (less tables > and mapping tables which all just duplicate each other). > I don't think this change is something that heavy that it's not worth > doing, and for me keeping it DRY is an advantage, just as you would > refactor code to do the same. > the advantage i see is when adding a feature to vm, and it need to be added in vm_static, it usually need also to be added to the template object as well, it will help remembering and make it easier.. (although it is not so common) > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > > > Regards, > Mike > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From iheim at redhat.com Tue Nov 15 23:55:52 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 16 Nov 2011 01:55:52 +0200 Subject: [Engine-devel] New version of git-request-review In-Reply-To: <4EBB9E8A.1060506@redhat.com> References: <4EBB9E8A.1060506@redhat.com> Message-ID: <4EC2FC08.5060806@redhat.com> On 11/10/2011 11:51 AM, Saggi Mizrahi wrote: > I just finished a version of my git-request-review script. how about a wiki page with this and the script as a file attachment, linked from here? http://www.ovirt.org/wiki/Working_with_oVirt_Gerrit > > For those who are not yet using it it's meant to make sending patches to > gerrit more streamlined. > It show you a list of the patches before sending them and also warns you > if you are not rebased. > > To have it working you need to have a configured remote for gerrit > already in your git config > In order for git to detect it as a new verb you have to have this in > your path > > usage: git request-review > > eg. git request-review gerrit master > > For those of you who are using the old version will probably notice this > version forgoes special gerrit configuration and can support multiple > gerrit servers (just configure more remotes). > > Fixes, features and comments are always welcome. > > Happy reviewing. > > > > > _______________________________________________ > vdsm-devel mailing list > vdsm-devel at lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/vdsm-devel From mburns at redhat.com Wed Nov 16 00:36:45 2011 From: mburns at redhat.com (Mike Burns) Date: Tue, 15 Nov 2011 19:36:45 -0500 Subject: [Engine-devel] New version of git-request-review In-Reply-To: <4EBB9E8A.1060506@redhat.com> References: <4EBB9E8A.1060506@redhat.com> Message-ID: <1321403805.2936.0.camel@mburns-laptop> On Thu, 2011-11-10 at 11:51 +0200, Saggi Mizrahi wrote: > I just finished a version of my git-request-review script. > > For those who are not yet using it it's meant to make sending patches > to gerrit more streamlined. > It show you a list of the patches before sending them and also warns > you if you are not rebased. > > To have it working you need to have a configured remote for gerrit > already in your git config > In order for git to detect it as a new verb you have to have this in > your path > > usage: git request-review > > eg. git request-review gerrit master > > For those of you who are using the old version will probably notice > this version forgoes special gerrit configuration and can support > multiple gerrit servers (just configure more remotes). > > Fixes, features and comments are always welcome. > > Happy reviewing. Very nice. Copying node-devel as well. This is much more convenient than manually typing the push command. Mike > > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From apevec at gmail.com Wed Nov 16 00:39:06 2011 From: apevec at gmail.com (Alan Pevec) Date: Wed, 16 Nov 2011 01:39:06 +0100 Subject: [Engine-devel] New version of git-request-review In-Reply-To: <4EC2FC08.5060806@redhat.com> References: <4EBB9E8A.1060506@redhat.com> <4EC2FC08.5060806@redhat.com> Message-ID: On Wed, Nov 16, 2011 at 12:55 AM, Itamar Heim wrote: > On 11/10/2011 11:51 AM, Saggi Mizrahi wrote: >> >> I just finished a version of my git-request-review script. > > how about a wiki page with this and the script as a file attachment, linked > from here? > http://www.ovirt.org/wiki/Working_with_oVirt_Gerrit > >> >> For those who are not yet using it it's meant to make sending patches to >> gerrit more streamlined. Have you looked at https://github.com/openstack-ci/git-review ? If nothing else, it might give us ideas how to optimize workflow with Gerrit. Openstack's flow is described in http://wiki.openstack.org/GerritWorkflow Alan From wangbo_bupt at hotmail.com Wed Nov 16 11:04:59 2011 From: wangbo_bupt at hotmail.com (hotmail) Date: Wed, 16 Nov 2011 19:04:59 +0800 Subject: [Engine-devel] About mix storage domains Message-ID: Hi! I want to know when will we can have a datacenter where we can mix storage domains?especially we can add more than one host into a datacenter with local storage domain. Thanks!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From smizrahi at redhat.com Wed Nov 16 12:25:18 2011 From: smizrahi at redhat.com (Saggi Mizrahi) Date: Wed, 16 Nov 2011 14:25:18 +0200 Subject: [Engine-devel] New version of git-request-review In-Reply-To: References: <4EBB9E8A.1060506@redhat.com> <4EC2FC08.5060806@redhat.com> Message-ID: <4EC3ABAE.8000909@redhat.com> On Wed 16 Nov 2011 02:39:06 AM IST, Alan Pevec wrote: > On Wed, Nov 16, 2011 at 12:55 AM, Itamar Heim wrote: >> On 11/10/2011 11:51 AM, Saggi Mizrahi wrote: >>> >>> I just finished a version of my git-request-review script. >> >> how about a wiki page with this and the script as a file attachment, linked >> from here? >> http://www.ovirt.org/wiki/Working_with_oVirt_Gerrit >> >>> >>> For those who are not yet using it it's meant to make sending patches to >>> gerrit more streamlined. > > Have you looked at https://github.com/openstack-ci/git-review ? > If nothing else, it might give us ideas how to optimize workflow with Gerrit. > Openstack's flow is described in http://wiki.openstack.org/GerritWorkflow > > Alan The open-stack scripts are superior and there is no reason not to join forces. I just switched to using them. I had issues with the output so If you like colors and branch\tag names in your output you can use the branch I posted for pull request. https://github.com/openstack-ci/git-review/pull/2 From mlipchuk at redhat.com Wed Nov 16 12:48:40 2011 From: mlipchuk at redhat.com (Maor) Date: Wed, 16 Nov 2011 14:48:40 +0200 Subject: [Engine-devel] Quota feature description In-Reply-To: <4EC3B07C.5010605@redhat.com> References: <4EC3B07C.5010605@redhat.com> Message-ID: <4EC3B128.20106@redhat.com> Hello all, The Quota feature description is published under the following links: http://www.ovirt.org/wiki/Features/DetailedQuota http://www.ovirt.org/wiki/Features/Quota Notice that screens of UI mockups should be updated. Please feel free, to share your comments about it. Thank you, Maor From wangbo at opzoon.com Wed Nov 16 10:57:05 2011 From: wangbo at opzoon.com (wangbo) Date: Wed, 16 Nov 2011 18:57:05 +0800 Subject: [Engine-devel] About mix storage domains Message-ID: <01f101cca44e$7a719450$6f54bcf0$@com> Hi! I want to know when will we can have a datacenter where we can mix storage domains?especially we can add more than one host into a datacenter with local storage domain. Thanks!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From iheim at redhat.com Wed Nov 16 15:49:41 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 16 Nov 2011 17:49:41 +0200 Subject: [Engine-devel] [Users] About mix storage domains In-Reply-To: References: Message-ID: <4EC3DB95.2070209@redhat.com> On 11/16/2011 01:04 PM, hotmail wrote: > Hi! > > I want to know when will we can have a datacenter where we can mix > storage domains?especially we can add more than one host into a > datacenter with local storage domain. plan is to remove the storage pool concept ("spm") with storage domain only ("sdm"). that's a major change in vdsm ongoing now, then need to change engine to match as well. will take some time to do this. From lpeer at redhat.com Thu Nov 17 07:08:13 2011 From: lpeer at redhat.com (Livnat Peer) Date: Thu, 17 Nov 2011 02:08:13 -0500 (EST) Subject: [Engine-devel] oVirt engine core Message-ID: The following is a new meeting request: Subject: oVirt engine core Organizer: "Livnat Peer" Time: 4:00:00 PM - 5:00:00 PM GMT +02:00 Jerusalem Recurrence : Every 2 week(s) on Wednesday No end date Effective Nov 23, 2011 Invitees: engine-devel at ovirt.org *~*~*~*~*~*~*~*~*~* This is a bi-weekly call about oVirt engine core development issues upstream. You are welcome to join. Bridge ID: 1814335863 Dial-in information: Reservationless-Plus Toll Free Dial-In Number (US & Canada): (800) 451-8679 Reservationless-Plus International Dial-In Number: (212) 729-5016 Conference code: 1814335863 Global Access Numbers Local: Australia, Sydney Dial-In #: 0289852326 Austria, Vienna Dial-In #: 012534978196 Belgium, Brussels Dial-In #: 027920405 China Dial-In #: 4006205013 Denmark, Copenhagen Dial-In #: 32729215 Finland, Helsinki Dial-In #: 0923194436 France, Paris Dial-In #: 0170377140 Germany, Berlin Dial-In #: 030300190579 Ireland, Dublin Dial-In #: 014367793 Italy, Milan Dial-In #: 0236269529 Netherlands, Amsterdam Dial-In #: 0207975872 Norway, Oslo Dial-In #: 21033188 Singapore Dial-In #: 64840858 Spain, Barcelona Dial-In #: 935452328 Sweden, Stockholm Dial-In #: 0850513770 Switzerland, Geneva Dial-In #: 0225927881 United Kingdom Dial-In #: 02078970515 United Kingdom Dial-In #: 08445790676 United Kingdom, LocalCall Dial-In #: 08445790678 United States Dial-In #: 2127295016 Global Access Numbers Tollfree: Argentina Dial-In #: 8004441016 Australia Dial-In #: 1800337169 Austria Dial-In #: 0800005898 Bahamas Dial-In #: 18002054776 Bahrain Dial-In #: 80004377 Belgium Dial-In #: 080048325 Brazil Dial-In #: 08008921002 Bulgaria Dial-In #: 008001100236 Chile Dial-In #: 800370228 Colombia Dial-In #: 018009134033 Costa Rica Dial-In #: 08000131048 Cyprus Dial-In #: 80095297 Czech Republic Dial-In #: 800700318 Denmark Dial-In #: 80887114 Dominican Republic Dial-In #: 18887512313 Estonia Dial-In #: 8000100232 Finland Dial-In #: 0800117116 France Dial-In #: 0805632867 Germany Dial-In #: 8006647541 Greece Dial-In #: 00800127562 Hong Kong Dial-In #: 800930349 Hungary Dial-In #: 0680016796 Iceland Dial-In #: 8008967 India Dial-In #: 0008006501533 Indonesia Dial-In #: 0018030179162 Ireland Dial-In #: 1800932401 Israel Dial-In #: 1809462557 Italy Dial-In #: 800985897 Jamaica Dial-In #: 18002050328 Japan Dial-In #: 0120934453 Korea (South) Dial-In #: 007986517393 Latvia Dial-In #: 80003339 Lithuania Dial-In #: 880030479 Luxembourg Dial-In #: 80026595 Malaysia Dial-In #: 1800814451 Mexico Dial-In #: 0018664590915 New Zealand Dial-In #: 0800888167 Norway Dial-In #: 80012994 Panama Dial-In #: 008002269184 Philippines Dial-In #: 180011100991 Poland Dial-In #: 008001210187 Portugal Dial-In #: 800814625 Russian Federation Dial-In #: 81080028341012 Saint Kitts and Nevis Dial-In #: 18002059252 Singapore Dial-In #: 8006162235 Slovak Republic Dial-In #: 0800001441 South Africa Dial-In #: 0800981148 Spain Dial-In #: 800300524 Sweden Dial-In #: 200896860 Switzerland Dial-In #: 800650077 Taiwan Dial-In #: 00801127141 Thailand Dial-In #: 001800656966 Trinidad and Tobago Dial-In #: 18002024615 United Arab Emirates Dial-In #: 8000650591 United Kingdom Dial-In #: 08006948057 United States Dial-In #: 8004518679 Uruguay Dial-In #: 00040190315 Venezuela Dial-In #: 08001627182 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: meeting.ics Type: text/calendar Size: 8680 bytes Desc: not available URL: From jfehlig at suse.com Wed Nov 16 23:51:46 2011 From: jfehlig at suse.com (Jim Fehlig) Date: Wed, 16 Nov 2011 16:51:46 -0700 Subject: [Engine-devel] problem trying to add a host In-Reply-To: <2036544639.2822245.1321357504278.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> References: <2036544639.2822245.1321357504278.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> Message-ID: <4EC44C92.5030208@suse.com> Doron Fediuck wrote: > Sent from my Android phone. Please ignore typos. > > -----Original Message----- > From: Duncan Mac-Vicar P. [dmacvicar at suse.de] > Received: Tuesday, 15 Nov 2011, 13:37 > To: engine-devel at ovirt.org > CC: Jim Fehlig [JFEHLIG at suse.com] > Subject: Re: [Engine-devel] problem trying to add a host > > On 11/15/2011 11:37 AM, Doron Fediuck wrote: > >> and password types (defined in sshd_config file). Please ensure you have (PasswordAuthentication yes) >> as well as (UsePAM yes). Keyboard-interactive type is unsupported currently. >> >> > > Thanks Doron. This fixed the issue and the host is now added!. > > Now I have two AIs > > (!) Power Management is not configured for this Host. > Enable Power Management > > (not a problem after I figure out all the options of that page) > > (!) NETWORK_UNREACHABLE > > Is this the connection from vdsm to the engine? Does it need a port open > in the opposite direction? > > You need to create a bridge called engine. Try and let us know. > Yep, after creating a bridge named "engine" on the host, the engine now sees the node as "Up". Before creating the bridge, I tried changing 'default_bridge' to an existing bridge (br0) in /etc/vdsm/vdsm.conf to no avail. Regards, Jim From smizrahi at redhat.com Thu Nov 17 08:52:35 2011 From: smizrahi at redhat.com (Saggi Mizrahi) Date: Thu, 17 Nov 2011 10:52:35 +0200 Subject: [Engine-devel] About mix storage domains In-Reply-To: <01f101cca44e$7a719450$6f54bcf0$@com> References: <01f101cca44e$7a719450$6f54bcf0$@com> Message-ID: <4EC4CB53.3020906@redhat.com> On Wed 16 Nov 2011 12:57:05 PM IST, wangbo wrote: > Hi! > > I want to know when will we can have a datacenter where we can mix > storage domains?especially we can add more than one host into a > datacenter with local storage domain. > > Thanks!! > > > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel Currently you cannot, we plan on removing this limitation but it'll have to be finalized in VDSM before it can propagate to management. This is months away though. From smizrahi at redhat.com Thu Nov 17 12:36:40 2011 From: smizrahi at redhat.com (Saggi Mizrahi) Date: Thu, 17 Nov 2011 14:36:40 +0200 Subject: [Engine-devel] (no subject) In-Reply-To: <> References: <4EC3B07C.5010605@redhat.com> <> Message-ID: <4EC4FFD8.6050402@redhat.com> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: > Hello all, > > The Quota feature description is published under the following links: > http://www.ovirt.org/wiki/Features/DetailedQuota > http://www.ovirt.org/wiki/Features/Quota > Notice that screens of UI mockups should be updated. > > Please feel free, to share your comments about it. > > Thank you, > Maor > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel I can't see how the host is supposed to enforce and handle it. Pause the VM? Crash it? raise ENOMEM\ENOSPC in the guest? Also what about cases of KSM\QCow2, disk\memory overcommit. Disk Templates. Storage for hibernation disk. Temporary and shared disks. Shared disks between VMs owned by different users. Backup snapshots (should they count in the quota? They are transient) I also don't see how vcpu limiting is any good? I don't even know what it means. What happens in migration between hosts with different amount of physical CPUs? I also don't think CPU limiting is even a thing to be managed by a quota. There is no reason not to use 100% of the CPU if you are the only VM active. CPU scheduling should use a priority model and not a quota IMHO. From mlipchuk at redhat.com Thu Nov 17 13:29:48 2011 From: mlipchuk at redhat.com (Maor) Date: Thu, 17 Nov 2011 15:29:48 +0200 Subject: [Engine-devel] Quota feature description In-Reply-To: <4EC4FFD8.6050402@redhat.com> References: <4EC3B07C.5010605@redhat.com> <> <4EC4FFD8.6050402@redhat.com> Message-ID: <4EC50C4C.90107@redhat.com> Hi Saggi, thanks for the comments, please see my comments in line On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: > On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: >> Hello all, >> >> The Quota feature description is published under the following links: >> http://www.ovirt.org/wiki/Features/DetailedQuota >> http://www.ovirt.org/wiki/Features/Quota >> Notice that screens of UI mockups should be updated. >> >> Please feel free, to share your comments about it. >> >> Thank you, >> Maor >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel > > I can't see how the host is supposed to enforce and handle it. Pause the > VM? Crash it? raise ENOMEM\ENOSPC in the guest? The enforcement and handling, should be from the engine scope and not from the Host perspective. Actually the Host, should not be aware of Quota at all. > Also what about cases of KSM\QCow2, disk\memory overcommit. on QCOW issue, the active disk should consume the full potential space from the Quota, since we are not sure how much space will be in use, slthough the snapshot disk, will be updated to consume only its real size from the Quota. you can check the Enforcement section : "When dealing with QCOW disks (which is not pre-allocated, like templates or stateless VM) the Quota should consume the total maximum size of the disk, since it is the potential size that can be used." for overcommit issue, please see CRUD section in the WIKI: "...However, users will not be able to exceed the Quota limitations again after the resources are released." > Disk Templates. > Storage for hibernation disk. > Temporary and shared disks. same logic as above (Enforcement section) > Shared disks between VMs owned by different users. Please see Dependencies / Related Features and Projects: "When handling plug/unplug disks or attach/detach disks, the entity will still consume resources from its configured original Quota it was created on. " Which means the disk should consume from the same Quota all the time (not dependent on the users that use it). > Backup snapshots (should they count in the quota? They are transient) When ever a volume is created whether it is snapshot, backup snapshot, stateless disk, or any QCOW implementation, the enforcement should the the same as described above (see Enforcement section) > > I also don't see how vcpu limiting is any good? I don't even know what > it means. What happens in migration between hosts with different amount > of physical CPUs? The "atomic" section that Quota is handling in the run time scope is cluster. Actually for the user migration will be transparent since it is consumed from the same Quota, the only validation the VM should encounter will be the same as before in the Host perspective. > I also don't think CPU limiting is even a thing to be managed by a > quota. There is no reason not to use 100% of the CPU if you are the only > VM active. CPU scheduling should use a priority model and not a quota IMHO. Again, the Quota should be managed from the engine level, and should not be reflected in the Host implementation. Try to look at it, as an abstract management mechanism for taking notes and managing resource consumes for the Administrator. A priority model is an interesting thought. Now it can be supported, by using different grace percentage from one Quota to another, or maybe create different Quota for Different type of users. From smizrahi at redhat.com Thu Nov 17 15:41:09 2011 From: smizrahi at redhat.com (Saggi Mizrahi) Date: Thu, 17 Nov 2011 17:41:09 +0200 Subject: [Engine-devel] Quota feature description In-Reply-To: <4EC50C4C.90107@redhat.com> References: <4EC3B07C.5010605@redhat.com> <> <4EC4FFD8.6050402@redhat.com> <4EC50C4C.90107@redhat.com> Message-ID: <4EC52B15.6000504@redhat.com> On Thu 17 Nov 2011 03:29:48 PM IST, Maor wrote: > Hi Saggi, thanks for the comments, please see my comments in line > > On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: >> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: >>> Hello all, >>> >>> The Quota feature description is published under the following links: >>> http://www.ovirt.org/wiki/Features/DetailedQuota >>> http://www.ovirt.org/wiki/Features/Quota >>> Notice that screens of UI mockups should be updated. >>> >>> Please feel free, to share your comments about it. >>> >>> Thank you, >>> Maor >>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >> >> I can't see how the host is supposed to enforce and handle it. Pause the >> VM? Crash it? raise ENOMEM\ENOSPC in the guest? > The enforcement and handling, should be from the engine scope and not > from the Host perspective. > Actually the Host, should not be aware of Quota at all. >> Also what about cases of KSM\QCow2, disk\memory overcommit. > on QCOW issue, the active disk should consume the full potential space > from the Quota, since we are not sure how much space will be in use, > slthough the snapshot disk, will be updated to consume only its real > size from the Quota. > > you can check the Enforcement section : > "When dealing with QCOW disks (which is not pre-allocated, like > templates or stateless VM) the Quota should consume the total maximum > size of the disk, since it is the potential size that can be used." > > for overcommit issue, please see CRUD section in the WIKI: > "...However, users will not be able to exceed the Quota limitations > again after the resources are released." > >> Disk Templates. >> Storage for hibernation disk. >> Temporary and shared disks. > same logic as above (Enforcement section) >> Shared disks between VMs owned by different users. > Please see Dependencies / Related Features and Projects: > "When handling plug/unplug disks or attach/detach disks, the entity will > still consume resources from its configured original Quota it was > created on. " > > Which means the disk should consume from the same Quota all the time > (not dependent on the users that use it). >> Backup snapshots (should they count in the quota? They are transient) > When ever a volume is created whether it is snapshot, backup snapshot, > stateless disk, or any QCOW implementation, the enforcement should the > the same as described above (see Enforcement section) >> >> I also don't see how vcpu limiting is any good? I don't even know what >> it means. What happens in migration between hosts with different amount >> of physical CPUs? > The "atomic" section that Quota is handling in the run time scope is > cluster. > Actually for the user migration will be transparent since it is consumed > from the same Quota, the only validation the VM should encounter will be > the same as before in the Host perspective. > >> I also don't think CPU limiting is even a thing to be managed by a >> quota. There is no reason not to use 100% of the CPU if you are the only >> VM active. CPU scheduling should use a priority model and not a quota IMHO. > Again, the Quota should be managed from the engine level, and should not > be reflected in the Host implementation. > Try to look at it, as an abstract management mechanism for taking notes > and managing resource consumes for the Administrator. > > A priority model is an interesting thought. > Now it can be supported, by using different grace percentage from one > Quota to another, or maybe create different Quota for Different type of > users. So I understand there is a pure disconnect between host lever quotas and policies and this quota feature. Speaking with Livnat and Maor I see this is a good use case for an ovirt user to allow simple quotas to clients. It just feels like a feature that should be implemented as part of a grater policy structure validating user actions and not as a strict quota. I think snapshots and images should have different quotas where images are capped by size and snapshots are capped by number. Think of it from a hosting provider perspective. I want each client to get a maximum of 30GB of storage. I can limit them to 30GB. And let them calculated that if they made a 16GB worth of VM they can never snapshot it. On the other hand I can chose to limit them to 10GB and allow up to 2 snapshots to each clients. This way I know each client will take up to 30 GB and there will be no complaints when the user created a VM and now he can't snapshot it. Snapshotting can also be capped on the image level instead of the user level (3 snapshots per image). I tend to go towards the latter because it'll keep multiple disk VM snapshots from failing due to depleted quota. This will also make complaints about the pessimistic storage estimation invalid. From mlipchuk at redhat.com Thu Nov 17 18:16:22 2011 From: mlipchuk at redhat.com (Maor) Date: Thu, 17 Nov 2011 20:16:22 +0200 Subject: [Engine-devel] Quota feature description In-Reply-To: <4EC52B15.6000504@redhat.com> References: <4EC3B07C.5010605@redhat.com> <> <4EC4FFD8.6050402@redhat.com> <4EC50C4C.90107@redhat.com> <4EC52B15.6000504@redhat.com> Message-ID: <4EC54F76.8000909@redhat.com> I updated the detailedQuota wiki, to sharpen that the Quota is managed from the engine-core perspective. Your suggestion about the number of snapshots per image is quite interesting, I think a PM should give an opinion about it. Although I'm not sure the limitation, should be in the Quota scope, maybe it is more accurate to set it as a limit in the VM scope. or the image scope instead. On 11/17/2011 05:41 PM, Saggi Mizrahi wrote: > On Thu 17 Nov 2011 03:29:48 PM IST, Maor wrote: >> Hi Saggi, thanks for the comments, please see my comments in line >> >> On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: >>> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: >>>> Hello all, >>>> >>>> The Quota feature description is published under the following links: >>>> http://www.ovirt.org/wiki/Features/DetailedQuota >>>> http://www.ovirt.org/wiki/Features/Quota >>>> Notice that screens of UI mockups should be updated. >>>> >>>> Please feel free, to share your comments about it. >>>> >>>> Thank you, >>>> Maor >>>> >>>> _______________________________________________ >>>> Engine-devel mailing list >>>> Engine-devel at ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>> >>> I can't see how the host is supposed to enforce and handle it. Pause the >>> VM? Crash it? raise ENOMEM\ENOSPC in the guest? >> The enforcement and handling, should be from the engine scope and not >> from the Host perspective. >> Actually the Host, should not be aware of Quota at all. >>> Also what about cases of KSM\QCow2, disk\memory overcommit. >> on QCOW issue, the active disk should consume the full potential space >> from the Quota, since we are not sure how much space will be in use, >> slthough the snapshot disk, will be updated to consume only its real >> size from the Quota. >> >> you can check the Enforcement section : >> "When dealing with QCOW disks (which is not pre-allocated, like >> templates or stateless VM) the Quota should consume the total maximum >> size of the disk, since it is the potential size that can be used." >> >> for overcommit issue, please see CRUD section in the WIKI: >> "...However, users will not be able to exceed the Quota limitations >> again after the resources are released." >> >>> Disk Templates. >>> Storage for hibernation disk. >>> Temporary and shared disks. >> same logic as above (Enforcement section) >>> Shared disks between VMs owned by different users. >> Please see Dependencies / Related Features and Projects: >> "When handling plug/unplug disks or attach/detach disks, the entity will >> still consume resources from its configured original Quota it was >> created on. " >> >> Which means the disk should consume from the same Quota all the time >> (not dependent on the users that use it). >>> Backup snapshots (should they count in the quota? They are transient) >> When ever a volume is created whether it is snapshot, backup snapshot, >> stateless disk, or any QCOW implementation, the enforcement should the >> the same as described above (see Enforcement section) >>> >>> I also don't see how vcpu limiting is any good? I don't even know what >>> it means. What happens in migration between hosts with different amount >>> of physical CPUs? >> The "atomic" section that Quota is handling in the run time scope is >> cluster. >> Actually for the user migration will be transparent since it is consumed >> from the same Quota, the only validation the VM should encounter will be >> the same as before in the Host perspective. >> >>> I also don't think CPU limiting is even a thing to be managed by a >>> quota. There is no reason not to use 100% of the CPU if you are the only >>> VM active. CPU scheduling should use a priority model and not a quota >>> IMHO. >> Again, the Quota should be managed from the engine level, and should not >> be reflected in the Host implementation. >> Try to look at it, as an abstract management mechanism for taking notes >> and managing resource consumes for the Administrator. >> >> A priority model is an interesting thought. >> Now it can be supported, by using different grace percentage from one >> Quota to another, or maybe create different Quota for Different type of >> users. > > So I understand there is a pure disconnect between host lever quotas and > policies and this quota feature. > Speaking with Livnat and Maor I see this is a good use case for an ovirt > user to allow simple quotas to clients. > > It just feels like a feature that should be implemented as part of a > grater policy structure validating user actions and not as a strict quota. > > > I think snapshots and images should have different quotas where images > are capped by size and snapshots are capped by number. > > Think of it from a hosting provider perspective. I want each client to > get a maximum of 30GB of storage. > I can limit them to 30GB. And let them calculated that if they made a > 16GB worth of VM they can never snapshot it. On the other hand I can > chose to limit them to 10GB and allow up to 2 snapshots to each clients. > This way I know each client will take up to 30 GB and there will be no > complaints when the user created a VM and now he can't snapshot it. > Snapshotting can also be capped on the image level instead of the user > level (3 snapshots per image). I tend to go towards the latter because > it'll keep multiple disk VM snapshots from failing due to depleted quota. > > This will also make complaints about the pessimistic storage estimation > invalid. From lhornyak at redhat.com Fri Nov 18 11:42:58 2011 From: lhornyak at redhat.com (Laszlo Hornyak) Date: Fri, 18 Nov 2011 06:42:58 -0500 (EST) Subject: [Engine-devel] logging in engine In-Reply-To: <913de2b3-aa3b-44b7-8d3d-2b82537bc751@zmail01.collab.prod.int.phx2.redhat.com> Message-ID: <701c2b1b-49a0-4ffa-9b3d-abde49d13edd@zmail01.collab.prod.int.phx2.redhat.com> Hi, In oVirt's logging configuration we direct org.ovirt.* loggers to the engine log appender, org.ovirt.engine.ui to the ui's log and then the rest to the server log. The difficulties come when you start to do cross-component debugging. E.g. you see the logs of the engine in the engine log, but you would also like to see the logs of the components the engine is building upon: persistence API's (jdbc, hibernate) network communication components (e.g. httpclient, vdsm, ssh client), data processing API's (xml or json-parsers, apache commons apis) and all these logs go to the generic engine log. So when debugging you would have to merge these logs, otherwise you may miss some important details. I believe Moti's patch (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) is one step in the another (I believe more useful) direction. But rather than specifying all the components that the engine uses, wouldn't it make sense to take just let all the logs flow into a single appender? What is your opinion? Thank you, Laszlo From jchoate at redhat.com Fri Nov 18 14:16:11 2011 From: jchoate at redhat.com (Jon Choate) Date: Fri, 18 Nov 2011 09:16:11 -0500 Subject: [Engine-devel] logging in engine In-Reply-To: <701c2b1b-49a0-4ffa-9b3d-abde49d13edd@zmail01.collab.prod.int.phx2.redhat.com> References: <701c2b1b-49a0-4ffa-9b3d-abde49d13edd@zmail01.collab.prod.int.phx2.redhat.com> Message-ID: <4EC668AB.2040301@redhat.com> On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > Hi, > > In oVirt's logging configuration we direct org.ovirt.* loggers to the engine log appender, org.ovirt.engine.ui to the ui's log and then the rest to the server log. > The difficulties come when you start to do cross-component debugging. E.g. you see the logs of the engine in the engine log, but you would also like to see the logs of the components the engine is building upon: persistence API's (jdbc, hibernate) network communication components (e.g. httpclient, vdsm, ssh client), data processing API's (xml or json-parsers, apache commons apis) and all these logs go to the generic engine log. So when debugging you would have to merge these logs, otherwise you may miss some important details. > > I believe Moti's patch (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) is one step in the another (I believe more useful) direction. > But rather than specifying all the components that the engine uses, wouldn't it make sense to take just let all the logs flow into a single appender? > > What is your opinion? > > Thank you, > Laszlo > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel Has any consideration been given to using the syslog appender and sending all messages there? Most data centers already have tools in place for analyzing the contents of syslog. From lhornyak at redhat.com Fri Nov 18 15:36:16 2011 From: lhornyak at redhat.com (Laszlo Hornyak) Date: Fri, 18 Nov 2011 10:36:16 -0500 (EST) Subject: [Engine-devel] logging in engine In-Reply-To: <4EC668AB.2040301@redhat.com> Message-ID: Yep, good question. Most Unix/linux sysadmins prefer syslog, whatever we have on the java+jboss side. ----- Original Message ----- > From: "Jon Choate" > To: engine-devel at ovirt.org > Sent: Friday, November 18, 2011 3:16:11 PM > Subject: Re: [Engine-devel] logging in engine > > On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > > Hi, > > > > In oVirt's logging configuration we direct org.ovirt.* loggers to > > the engine log appender, org.ovirt.engine.ui to the ui's log and > > then the rest to the server log. > > The difficulties come when you start to do cross-component > > debugging. E.g. you see the logs of the engine in the engine log, > > but you would also like to see the logs of the components the > > engine is building upon: persistence API's (jdbc, hibernate) > > network communication components (e.g. httpclient, vdsm, ssh > > client), data processing API's (xml or json-parsers, apache > > commons apis) and all these logs go to the generic engine log. So > > when debugging you would have to merge these logs, otherwise you > > may miss some important details. > > > > I believe Moti's patch > > (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) > > is one step in the another (I believe more useful) direction. > > But rather than specifying all the components that the engine uses, > > wouldn't it make sense to take just let all the logs flow into a > > single appender? > > > > What is your opinion? > > > > Thank you, > > Laszlo > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > Has any consideration been given to using the syslog appender and > sending all messages there? Most data centers already have tools in > place for analyzing the contents of syslog. > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From yzaslavs at redhat.com Sun Nov 20 08:45:40 2011 From: yzaslavs at redhat.com (Yair Zaslavsky) Date: Sun, 20 Nov 2011 10:45:40 +0200 Subject: [Engine-devel] Please run the upgrade script Message-ID: <4EC8BE34.4090206@redhat.com> Hi all, After you fetch and rebase, please run the db upgrade script located at ovrit_engine/backend/manager/dbscripts Kind regards, Yair From lpeer at redhat.com Sun Nov 20 09:29:34 2011 From: lpeer at redhat.com (Livnat Peer) Date: Sun, 20 Nov 2011 11:29:34 +0200 Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <4EC8BE34.4090206@redhat.com> References: <4EC8BE34.4090206@redhat.com> Message-ID: <4EC8C87E.2010405@redhat.com> On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: > Hi all, > After you fetch and rebase, please run the db upgrade script located at > > ovrit_engine/backend/manager/dbscripts > > Kind regards, > > Yair Hi Yair, I think that running db upgrade script should be part of the routine procedure of getting the latest code. The upgrading script is re-entrant and should not cause issues if using it on any fetch. I added "getting latest" section in the wiki - http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest Thanks, Livnat From emesika at redhat.com Sun Nov 20 11:35:24 2011 From: emesika at redhat.com (Eli Mesika) Date: Sun, 20 Nov 2011 06:35:24 -0500 (EST) Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <4EC8C87E.2010405@redhat.com> Message-ID: <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > From: "Livnat Peer" > To: "Yair Zaslavsky" > Cc: engine-devel at ovirt.org > Sent: Sunday, November 20, 2011 11:29:34 AM > Subject: Re: [Engine-devel] Please run the upgrade script > > On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: > > Hi all, > > After you fetch and rebase, please run the db upgrade script > > located at > > > > ovrit_engine/backend/manager/dbscripts > > > > Kind regards, > > > > Yair > > Hi Yair, > > I think that running db upgrade script should be part of the routine > procedure of getting the latest code. > The upgrading script is re-entrant and should not cause issues if > using > it on any fetch. right, actually even right now we can automate it from maven using the Maven EXEC plugin http://mojo.codehaus.org/exec-maven-plugin/ I have thought about it as a part of Flyway integration. > > I added "getting latest" section in the wiki - > http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest > > > Thanks, Livnat > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From rgolan at redhat.com Sun Nov 20 11:37:48 2011 From: rgolan at redhat.com (Roy Golan) Date: Sun, 20 Nov 2011 13:37:48 +0200 Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> References: <4EC8C87E.2010405@redhat.com> <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> Message-ID: <4EC8E68C.9000408@redhat.com> On Sun 20 Nov 2011 01:35:24 PM IST, Eli Mesika wrote: > > > ----- Original Message ----- >> From: "Livnat Peer" >> To: "Yair Zaslavsky" >> Cc: engine-devel at ovirt.org >> Sent: Sunday, November 20, 2011 11:29:34 AM >> Subject: Re: [Engine-devel] Please run the upgrade script >> >> On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: >>> Hi all, >>> After you fetch and rebase, please run the db upgrade script >>> located at >>> >>> ovrit_engine/backend/manager/dbscripts >>> >>> Kind regards, >>> >>> Yair >> >> Hi Yair, >> >> I think that running db upgrade script should be part of the routine >> procedure of getting the latest code. >> The upgrading script is re-entrant and should not cause issues if >> using >> it on any fetch. > > right, actually even right now we can automate it from maven using the Maven EXEC plugin > http://mojo.codehaus.org/exec-maven-plugin/ > > I have thought about it as a part of Flyway integration. > +1 for upgrading the db using maven > >> >> I added "getting latest" section in the wiki - >> http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest >> >> >> Thanks, Livnat >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From dfediuck at redhat.com Sun Nov 20 12:35:53 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Sun, 20 Nov 2011 14:35:53 +0200 Subject: [Engine-devel] logging in engine In-Reply-To: References: Message-ID: <201111201435.54110.dfediuck@redhat.com> On Friday 18 November 2011 17:36:16 Laszlo Hornyak wrote: > Yep, good question. Most Unix/linux sysadmins prefer syslog, whatever we have on the java+jboss side. > > ----- Original Message ----- > > From: "Jon Choate" > > To: engine-devel at ovirt.org > > Sent: Friday, November 18, 2011 3:16:11 PM > > Subject: Re: [Engine-devel] logging in engine > > > > On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > > > Hi, > > > > > > In oVirt's logging configuration we direct org.ovirt.* loggers to > > > the engine log appender, org.ovirt.engine.ui to the ui's log and > > > then the rest to the server log. > > > The difficulties come when you start to do cross-component > > > debugging. E.g. you see the logs of the engine in the engine log, > > > but you would also like to see the logs of the components the > > > engine is building upon: persistence API's (jdbc, hibernate) > > > network communication components (e.g. httpclient, vdsm, ssh > > > client), data processing API's (xml or json-parsers, apache > > > commons apis) and all these logs go to the generic engine log. So > > > when debugging you would have to merge these logs, otherwise you > > > may miss some important details. > > > > > > I believe Moti's patch > > > (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) > > > is one step in the another (I believe more useful) direction. > > > But rather than specifying all the components that the engine uses, > > > wouldn't it make sense to take just let all the logs flow into a > > > single appender? > > > > > > What is your opinion? > > > > > > Thank you, > > > Laszlo > > > _______________________________________________ > > > Engine-devel mailing list > > > Engine-devel at ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > Has any consideration been given to using the syslog appender and > > sending all messages there? Most data centers already have tools in > > place for analyzing the contents of syslog. 1. We have a specific log collector tool to collect the relevant logs and configurations from the setup, since this is not a simple setup. We have hosts we'd like to monitor, core dumps, etc. So this is not a standalone machine debugging / logging which may use syslog. 2. Moreover, Java's logging allows you to control log verbosity at runtime. This is a feature we'd like to preserve, and not re-create all namespaces as filters in syslog... -- /d "Who's General Failure and why's he reading my disk?" From mlipchuk at redhat.com Sun Nov 20 12:46:48 2011 From: mlipchuk at redhat.com (Maor) Date: Sun, 20 Nov 2011 14:46:48 +0200 Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> References: <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> Message-ID: <4EC8F6B8.5070604@redhat.com> On 11/20/2011 01:35 PM, Eli Mesika wrote: > > > ----- Original Message ----- >> From: "Livnat Peer" >> To: "Yair Zaslavsky" >> Cc: engine-devel at ovirt.org >> Sent: Sunday, November 20, 2011 11:29:34 AM >> Subject: Re: [Engine-devel] Please run the upgrade script >> >> On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: >>> Hi all, >>> After you fetch and rebase, please run the db upgrade script >>> located at >>> >>> ovrit_engine/backend/manager/dbscripts >>> >>> Kind regards, >>> >>> Yair >> >> Hi Yair, >> >> I think that running db upgrade script should be part of the routine >> procedure of getting the latest code. >> The upgrading script is re-entrant and should not cause issues if >> using >> it on any fetch. > > right, actually even right now we can automate it from maven using the Maven EXEC plugin > http://mojo.codehaus.org/exec-maven-plugin/ > > I have thought about it as a part of Flyway integration. > > +1 For upgrading DB automatically. I think, that if we plan to update the DB from the maven build command, we should also take in consider adding a parameter indicating when we prefer not to run the upgrade script at each build (same as skipTests). Since after first git fetch, DB should not be changed. >> >> I added "getting latest" section in the wiki - >> http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest >> >> >> Thanks, Livnat >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From ofrenkel at redhat.com Sun Nov 20 12:47:49 2011 From: ofrenkel at redhat.com (Omer Frenkel) Date: Sun, 20 Nov 2011 07:47:49 -0500 (EST) Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <4EC8C87E.2010405@redhat.com> Message-ID: ----- Original Message ----- > From: "Livnat Peer" > To: "Yair Zaslavsky" > Cc: engine-devel at ovirt.org > Sent: Sunday, November 20, 2011 11:29:34 AM > Subject: Re: [Engine-devel] Please run the upgrade script > > On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: > > Hi all, > > After you fetch and rebase, please run the db upgrade script > > located at > > > > ovrit_engine/backend/manager/dbscripts > > > > Kind regards, > > > > Yair > > Hi Yair, > > I think that running db upgrade script should be part of the routine > procedure of getting the latest code. > The upgrading script is re-entrant and should not cause issues if > using > it on any fetch. > > I added "getting latest" section in the wiki - > http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest cool, shouldn't it be there: $> cd $OVIRT_HOME/backend/manager/dbscripts/ $> ./upgrade.sh -u postgres (at least until we change it to use maven) > > > Thanks, Livnat > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From lpeer at redhat.com Sun Nov 20 13:02:14 2011 From: lpeer at redhat.com (Livnat Peer) Date: Sun, 20 Nov 2011 15:02:14 +0200 Subject: [Engine-devel] Please run the upgrade script In-Reply-To: References: Message-ID: <4EC8FA56.6020703@redhat.com> On 11/20/2011 02:47 PM, Omer Frenkel wrote: > > > ----- Original Message ----- >> From: "Livnat Peer" >> To: "Yair Zaslavsky" >> Cc: engine-devel at ovirt.org >> Sent: Sunday, November 20, 2011 11:29:34 AM >> Subject: Re: [Engine-devel] Please run the upgrade script >> >> On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: >>> Hi all, >>> After you fetch and rebase, please run the db upgrade script >>> located at >>> >>> ovrit_engine/backend/manager/dbscripts >>> >>> Kind regards, >>> >>> Yair >> >> Hi Yair, >> >> I think that running db upgrade script should be part of the routine >> procedure of getting the latest code. >> The upgrading script is re-entrant and should not cause issues if >> using >> it on any fetch. >> >> I added "getting latest" section in the wiki - >> http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest > > cool, > shouldn't it be there: > > $> cd $OVIRT_HOME/backend/manager/dbscripts/ > $> ./upgrade.sh -u postgres > > (at least until we change it to use maven) > It is there, until adding a profile and updating it again :) >> >> >> Thanks, Livnat >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> From masayag at redhat.com Sun Nov 20 13:03:29 2011 From: masayag at redhat.com (Moti Asayag) Date: Sun, 20 Nov 2011 15:03:29 +0200 Subject: [Engine-devel] Please run the upgrade script In-Reply-To: <4EC8F6B8.5070604@redhat.com> References: <53a60ebb-03b0-4eb7-a004-d624c680b62f@zmail13.collab.prod.int.phx2.redhat.com> <4EC8F6B8.5070604@redhat.com> Message-ID: <4EC8FAA1.1040009@redhat.com> On 11/20/2011 02:46 PM, Maor wrote: > On 11/20/2011 01:35 PM, Eli Mesika wrote: >> >> >> ----- Original Message ----- >>> From: "Livnat Peer" >>> To: "Yair Zaslavsky" >>> Cc: engine-devel at ovirt.org >>> Sent: Sunday, November 20, 2011 11:29:34 AM >>> Subject: Re: [Engine-devel] Please run the upgrade script >>> >>> On 11/20/2011 10:45 AM, Yair Zaslavsky wrote: >>>> Hi all, >>>> After you fetch and rebase, please run the db upgrade script >>>> located at >>>> >>>> ovrit_engine/backend/manager/dbscripts >>>> >>>> Kind regards, >>>> >>>> Yair >>> >>> Hi Yair, >>> >>> I think that running db upgrade script should be part of the routine >>> procedure of getting the latest code. >>> The upgrading script is re-entrant and should not cause issues if >>> using >>> it on any fetch. >> >> right, actually even right now we can automate it from maven using the Maven EXEC plugin >> http://mojo.codehaus.org/exec-maven-plugin/ >> >> I have thought about it as a part of Flyway integration. >> >> > +1 For upgrading DB automatically. > > I think, that if we plan to update the DB from the maven build command, > we should also take in consider adding a parameter indicating when we > prefer not to run the upgrade script at each build (same as skipTests). > Since after first git fetch, DB should not be changed. In addition, there is need to handle dao and bll schemas update under the assumption they are differ from the default 'schema'. >>> >>> I added "getting latest" section in the wiki - >>> http://www.ovirt.org/wiki/Building_oVirt_engine#Getting_Latest >>> >>> >>> Thanks, Livnat >>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From ranglust at redhat.com Sun Nov 20 16:03:32 2011 From: ranglust at redhat.com (Ronen Angluster) Date: Sun, 20 Nov 2011 18:03:32 +0200 Subject: [Engine-devel] Ovirt/VDSM installation Status Message-ID: <4EC924D4.8040107@redhat.com> Hello, We successfully installed Ovirt-engine & VDSM (on the same host!) from the ovirt's repository. The Steps were as following: 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM package containing all the needed information) 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" 3. disable NetworkManager and activating the network service 4. create a bridge interface named "engine" 5. change configuration in /etc/libvirt/qeum.conf /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf 6. configure postgresql for first time installation 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to 'pc-0.14' (known bug in ovirt-engine, patch already submitted) 9. start vdsm & jboss services After those steps were carried, we were able to: 1. add new data center, cluster & host 2. configure storage 3. create and run a new VM a wiki page containing all the necessary information will be delivered tomorrow. Ronen From oschreib at redhat.com Sun Nov 20 16:04:34 2011 From: oschreib at redhat.com (Ofer Schreiber) Date: Sun, 20 Nov 2011 11:04:34 -0500 (EST) Subject: [Engine-devel] Ovirt/VDSM installation Status In-Reply-To: <4EC924D4.8040107@redhat.com> References: <4EC924D4.8040107@redhat.com> Message-ID: <021b01cca79e$1745bf20$45d13d60$@redhat.com> Some prof (F16 screen shot) attached. Ofer. > -----Original Message----- > From: Ronen Angluster [mailto:ranglust at redhat.com] > Sent: 20 November 2011 18:04 > To: Itamar Heim; Barak Azulay; Ofer Schreiber > Cc: Doron Fediuck; engine-devel at ovirt.org; Ayal Baron > Subject: Ovirt/VDSM installation Status > > Hello, > > We successfully installed Ovirt-engine & VDSM (on the same host!) from > the ovirt's repository. > > The Steps were as following: > 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM > package containing all the needed information) > 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" > 3. disable NetworkManager and activating the network service > 4. create a bridge interface named "engine" > 5. change configuration in /etc/libvirt/qeum.conf > /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf > 6. configure postgresql for first time installation > 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh > 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to > 'pc-0.14' (known bug in ovirt-engine, patch already submitted) > 9. start vdsm & jboss services > > After those steps were carried, we were able to: > 1. add new data center, cluster & host > 2. configure storage > 3. create and run a new VM > > a wiki page containing all the necessary information will be delivered > tomorrow. > > Ronen -------------- next part -------------- A non-text attachment was scrubbed... Name: oVirt.png Type: image/png Size: 118325 bytes Desc: not available URL: From dfediuck at redhat.com Sun Nov 20 16:23:47 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Sun, 20 Nov 2011 18:23:47 +0200 Subject: [Engine-devel] Ovirt/VDSM installation Status In-Reply-To: <021b01cca79e$1745bf20$45d13d60$@redhat.com> References: <4EC924D4.8040107@redhat.com> <021b01cca79e$1745bf20$45d13d60$@redhat.com> Message-ID: <201111201823.47225.dfediuck@redhat.com> Good job, guys! On Sunday 20 November 2011 18:04:34 Ofer Schreiber wrote: > Some prof (F16 screen shot) attached. > > Ofer. > > > -----Original Message----- > > From: Ronen Angluster [mailto:ranglust at redhat.com] > > Sent: 20 November 2011 18:04 > > To: Itamar Heim; Barak Azulay; Ofer Schreiber > > Cc: Doron Fediuck; engine-devel at ovirt.org; Ayal Baron > > Subject: Ovirt/VDSM installation Status > > > > Hello, > > > > We successfully installed Ovirt-engine & VDSM (on the same host!) from > > the ovirt's repository. > > > > The Steps were as following: > > 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM > > package containing all the needed information) > > 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" > > 3. disable NetworkManager and activating the network service > > 4. create a bridge interface named "engine" > > 5. change configuration in /etc/libvirt/qeum.conf > > /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf > > 6. configure postgresql for first time installation > > 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh > > 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to > > 'pc-0.14' (known bug in ovirt-engine, patch already submitted) > > 9. start vdsm & jboss services > > > > After those steps were carried, we were able to: > > 1. add new data center, cluster & host > > 2. configure storage > > 3. create and run a new VM > > > > a wiki page containing all the necessary information will be delivered > > tomorrow. > > > > Ronen > > -- /d ?Funny,? he intoned funereally, ?how just when you think life can't possibly get any worse it suddenly does.? --Douglas Adams, The Hitchhiker's Guide to the Galaxy From jchoate at redhat.com Mon Nov 21 02:12:57 2011 From: jchoate at redhat.com (Jon Choate) Date: Sun, 20 Nov 2011 21:12:57 -0500 (EST) Subject: [Engine-devel] logging in engine In-Reply-To: <201111201435.54110.dfediuck@redhat.com> Message-ID: ----- Original Message ----- > From: "Doron Fediuck" > To: engine-devel at ovirt.org > Cc: "Laszlo Hornyak" , "Jon Choate" > Sent: Sunday, November 20, 2011 7:35:53 AM > Subject: Re: [Engine-devel] logging in engine > > On Friday 18 November 2011 17:36:16 Laszlo Hornyak wrote: > > Yep, good question. Most Unix/linux sysadmins prefer syslog, > > whatever we have on the java+jboss side. > > > > ----- Original Message ----- > > > From: "Jon Choate" > > > To: engine-devel at ovirt.org > > > Sent: Friday, November 18, 2011 3:16:11 PM > > > Subject: Re: [Engine-devel] logging in engine > > > > > > On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > > > > Hi, > > > > > > > > In oVirt's logging configuration we direct org.ovirt.* loggers > > > > to > > > > the engine log appender, org.ovirt.engine.ui to the ui's log > > > > and > > > > then the rest to the server log. > > > > The difficulties come when you start to do cross-component > > > > debugging. E.g. you see the logs of the engine in the engine > > > > log, > > > > but you would also like to see the logs of the components the > > > > engine is building upon: persistence API's (jdbc, hibernate) > > > > network communication components (e.g. httpclient, vdsm, ssh > > > > client), data processing API's (xml or json-parsers, apache > > > > commons apis) and all these logs go to the generic engine log. > > > > So > > > > when debugging you would have to merge these logs, otherwise > > > > you > > > > may miss some important details. > > > > > > > > I believe Moti's patch > > > > (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) > > > > is one step in the another (I believe more useful) direction. > > > > But rather than specifying all the components that the engine > > > > uses, > > > > wouldn't it make sense to take just let all the logs flow into > > > > a > > > > single appender? > > > > > > > > What is your opinion? > > > > > > > > Thank you, > > > > Laszlo > > > > _______________________________________________ > > > > Engine-devel mailing list > > > > Engine-devel at ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > Has any consideration been given to using the syslog appender and > > > sending all messages there? Most data centers already have tools > > > in > > > place for analyzing the contents of syslog. > > 1. We have a specific log collector tool to collect the relevant logs > and configurations > from the setup, since this is not a simple setup. We have hosts we'd > like to monitor, > core dumps, etc. So this is not a standalone machine debugging / > logging which may > use syslog. That sounds like we've recreated our own syslog since syslog aggregates logs from several different applications and across different machines. > 2. Moreover, Java's logging allows you to control log verbosity at > runtime. This is > a feature we'd like to preserve, and not re-create all namespaces as > filters in syslog... > Log4J has a syslog appender so switching to syslog would require no code changes other than injecting a new appender. You would still be able to tweak verbosity at runtime. Syslog would also allow you to tee the messages to both a syslog for forwarding to a syslog aggregator and to an application specific log file on the local machine. Moreover, syslog is something our users (sysadmins) are comfortable and familiar with and have often built data center monitoring solutions around. > > -- > > /d > > "Who's General Failure and why's he reading my disk?" > From dfediuck at redhat.com Mon Nov 21 10:48:34 2011 From: dfediuck at redhat.com (Doron Fediuck) Date: Mon, 21 Nov 2011 12:48:34 +0200 Subject: [Engine-devel] logging in engine In-Reply-To: References: Message-ID: <201111211248.34381.dfediuck@redhat.com> On Monday 21 November 2011 04:12:57 Jon Choate wrote: > > ----- Original Message ----- > > From: "Doron Fediuck" > > To: engine-devel at ovirt.org > > Cc: "Laszlo Hornyak" , "Jon Choate" > > Sent: Sunday, November 20, 2011 7:35:53 AM > > Subject: Re: [Engine-devel] logging in engine > > > > On Friday 18 November 2011 17:36:16 Laszlo Hornyak wrote: > > > Yep, good question. Most Unix/linux sysadmins prefer syslog, > > > whatever we have on the java+jboss side. > > > > > > ----- Original Message ----- > > > > From: "Jon Choate" > > > > To: engine-devel at ovirt.org > > > > Sent: Friday, November 18, 2011 3:16:11 PM > > > > Subject: Re: [Engine-devel] logging in engine > > > > > > > > On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > > > > > Hi, > > > > > > > > > > In oVirt's logging configuration we direct org.ovirt.* loggers > > > > > to > > > > > the engine log appender, org.ovirt.engine.ui to the ui's log > > > > > and > > > > > then the rest to the server log. > > > > > The difficulties come when you start to do cross-component > > > > > debugging. E.g. you see the logs of the engine in the engine > > > > > log, > > > > > but you would also like to see the logs of the components the > > > > > engine is building upon: persistence API's (jdbc, hibernate) > > > > > network communication components (e.g. httpclient, vdsm, ssh > > > > > client), data processing API's (xml or json-parsers, apache > > > > > commons apis) and all these logs go to the generic engine log. > > > > > So > > > > > when debugging you would have to merge these logs, otherwise > > > > > you > > > > > may miss some important details. > > > > > > > > > > I believe Moti's patch > > > > > (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) > > > > > is one step in the another (I believe more useful) direction. > > > > > But rather than specifying all the components that the engine > > > > > uses, > > > > > wouldn't it make sense to take just let all the logs flow into > > > > > a > > > > > single appender? > > > > > > > > > > What is your opinion? > > > > > > > > > > Thank you, > > > > > Laszlo > > > > > _______________________________________________ > > > > > Engine-devel mailing list > > > > > Engine-devel at ovirt.org > > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > > Has any consideration been given to using the syslog appender and > > > > sending all messages there? Most data centers already have tools > > > > in > > > > place for analyzing the contents of syslog. > > > > 1. We have a specific log collector tool to collect the relevant logs > > and configurations > > from the setup, since this is not a simple setup. We have hosts we'd > > like to monitor, > > core dumps, etc. So this is not a standalone machine debugging / > > logging which may > > use syslog. > > That sounds like we've recreated our own syslog since syslog aggregates logs from several different applications and across different machines. > Not exactly, we're collecting configurations and coredumps as well. This is not only logging data. > > 2. Moreover, Java's logging allows you to control log verbosity at > > runtime. This is > > a feature we'd like to preserve, and not re-create all namespaces as > > filters in syslog... > > > > Log4J has a syslog appender so switching to syslog would require no code changes other than injecting a new appender. You would still be able to tweak verbosity at runtime. Syslog would also allow you to tee the messages to both a syslog for forwarding to a syslog aggregator and to an application specific log file on the local machine. > > Moreover, syslog is something our users (sysadmins) are comfortable and familiar with and have often built data center monitoring solutions around. > This may cause flooding of the syslog, filling it with messages which may mask host important issues such as hardware issues, etc. So you can create a filter on message source (facility) and direct it to a specific file, then you end up with what we already have.... To make is simple, if someone needs it, he can submit a patch with an alternate solution (jboss-log4j.xml.syslog), and the relevant changes on /etc/syslog.conf, and those who need it can use it. -- /d "Who's General Failure and why's he reading my disk?" From jchoate at redhat.com Mon Nov 21 12:32:44 2011 From: jchoate at redhat.com (Jon Choate) Date: Mon, 21 Nov 2011 07:32:44 -0500 (EST) Subject: [Engine-devel] logging in engine In-Reply-To: <201111211248.34381.dfediuck@redhat.com> Message-ID: Got it. Maybe the long term solution will be some nice person making an oVirt plugin for Nagios, Splunk etc. As I think about it I realize its better to defer it until we have a real requirement for it anyway. ----- Original Message ----- > From: "Doron Fediuck" > To: "Jon Choate" > Cc: "Laszlo Hornyak" , engine-devel at ovirt.org > Sent: Monday, November 21, 2011 5:48:34 AM > Subject: Re: [Engine-devel] logging in engine > > On Monday 21 November 2011 04:12:57 Jon Choate wrote: > > > > ----- Original Message ----- > > > From: "Doron Fediuck" > > > To: engine-devel at ovirt.org > > > Cc: "Laszlo Hornyak" , "Jon Choate" > > > > > > Sent: Sunday, November 20, 2011 7:35:53 AM > > > Subject: Re: [Engine-devel] logging in engine > > > > > > On Friday 18 November 2011 17:36:16 Laszlo Hornyak wrote: > > > > Yep, good question. Most Unix/linux sysadmins prefer syslog, > > > > whatever we have on the java+jboss side. > > > > > > > > ----- Original Message ----- > > > > > From: "Jon Choate" > > > > > To: engine-devel at ovirt.org > > > > > Sent: Friday, November 18, 2011 3:16:11 PM > > > > > Subject: Re: [Engine-devel] logging in engine > > > > > > > > > > On 11/18/2011 06:42 AM, Laszlo Hornyak wrote: > > > > > > Hi, > > > > > > > > > > > > In oVirt's logging configuration we direct org.ovirt.* > > > > > > loggers > > > > > > to > > > > > > the engine log appender, org.ovirt.engine.ui to the ui's > > > > > > log > > > > > > and > > > > > > then the rest to the server log. > > > > > > The difficulties come when you start to do cross-component > > > > > > debugging. E.g. you see the logs of the engine in the > > > > > > engine > > > > > > log, > > > > > > but you would also like to see the logs of the components > > > > > > the > > > > > > engine is building upon: persistence API's (jdbc, > > > > > > hibernate) > > > > > > network communication components (e.g. httpclient, vdsm, > > > > > > ssh > > > > > > client), data processing API's (xml or json-parsers, apache > > > > > > commons apis) and all these logs go to the generic engine > > > > > > log. > > > > > > So > > > > > > when debugging you would have to merge these logs, > > > > > > otherwise > > > > > > you > > > > > > may miss some important details. > > > > > > > > > > > > I believe Moti's patch > > > > > > (http://gerrit.ovirt.org/#patch,sidebyside,257,3,backend/manager/conf/jboss-log4j.xml) > > > > > > is one step in the another (I believe more useful) > > > > > > direction. > > > > > > But rather than specifying all the components that the > > > > > > engine > > > > > > uses, > > > > > > wouldn't it make sense to take just let all the logs flow > > > > > > into > > > > > > a > > > > > > single appender? > > > > > > > > > > > > What is your opinion? > > > > > > > > > > > > Thank you, > > > > > > Laszlo > > > > > > _______________________________________________ > > > > > > Engine-devel mailing list > > > > > > Engine-devel at ovirt.org > > > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > > > Has any consideration been given to using the syslog appender > > > > > and > > > > > sending all messages there? Most data centers already have > > > > > tools > > > > > in > > > > > place for analyzing the contents of syslog. > > > > > > 1. We have a specific log collector tool to collect the relevant > > > logs > > > and configurations > > > from the setup, since this is not a simple setup. We have hosts > > > we'd > > > like to monitor, > > > core dumps, etc. So this is not a standalone machine debugging / > > > logging which may > > > use syslog. > > > > That sounds like we've recreated our own syslog since syslog > > aggregates logs from several different applications and across > > different machines. > > > Not exactly, we're collecting configurations and coredumps as well. > This is not only logging data. > > > > 2. Moreover, Java's logging allows you to control log verbosity > > > at > > > runtime. This is > > > a feature we'd like to preserve, and not re-create all namespaces > > > as > > > filters in syslog... > > > > > > > Log4J has a syslog appender so switching to syslog would require no > > code changes other than injecting a new appender. You would still > > be able to tweak verbosity at runtime. Syslog would also allow you > > to tee the messages to both a syslog for forwarding to a syslog > > aggregator and to an application specific log file on the local > > machine. > > > > Moreover, syslog is something our users (sysadmins) are comfortable > > and familiar with and have often built data center monitoring > > solutions around. > > > This may cause flooding of the syslog, filling it with messages which > may mask host important > issues such as hardware issues, etc. So you can create a filter on > message source (facility) > and direct it to a specific file, then you end up with what we > already have.... > > To make is simple, if someone needs it, he can submit a patch with an > alternate solution (jboss-log4j.xml.syslog), > and the relevant changes on /etc/syslog.conf, and those who need it > can use it. > -- > > /d > > "Who's General Failure and why's he reading my disk?" > From lpeer at redhat.com Mon Nov 21 13:24:22 2011 From: lpeer at redhat.com (Livnat Peer) Date: Mon, 21 Nov 2011 15:24:22 +0200 Subject: [Engine-devel] engine-core bi-weekly agenda Message-ID: <4ECA5106.5020600@redhat.com> Hi All, Agenda for the core-engine meeting on Wednesday: * Building oVirt-engine * Maor Lipchuk will present the Quota feature. http://www.ovirt.org/wiki/Features/Quota http://www.ovirt.org/wiki/Features/DetailedQuota If you have more items you want to discuss please send to list and we'll add them to the agenda. Thank, Livnat From mpastern at redhat.com Mon Nov 21 14:28:48 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Mon, 21 Nov 2011 16:28:48 +0200 Subject: [Engine-devel] Classifying engine errors Message-ID: <4ECA6020.7090302@redhat.com> Hi, i would like to suggest classifying engine errors (VdcBllMessages) using these categories: 100-199. entity not found 200-300. missing parameters 300-399. no resources available 400-499. can't perform - user error 500-599. can't perform - server error 600-699. Unauthorized 700-x. Forbidden this way rest api will be able mapping them to HTTP errors e.g any error returned by engine CAN_DO_ACTION 100 < x < 200 will be converted to HTTP 404 with CAN_DO_ACTION message detail - this can be easily done by assigning id to each VdcBllMessages enum member in range of one of mentioned categories. -- Michael Pasternak RedHat, ENG-Virtualization R&D -- Michael Pasternak RedHat, ENG-Virtualization R&D From mpastern at redhat.com Mon Nov 21 14:47:09 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Mon, 21 Nov 2011 16:47:09 +0200 Subject: [Engine-devel] Fwd: Re: Classifying engine errors Message-ID: <4ECA646D.6090105@redhat.com> -------- Original Message -------- Subject: Re: Classifying engine errors Date: Mon, 21 Nov 2011 09:32:10 -0500 (EST) From: Jon Choate To: Michael Pasternak ----- Original Message ----- > From: "Michael Pasternak" > Sent: Monday, November 21, 2011 9:10:46 AM > Subject: Classifying engine errors > > > Hi, > > i would like to suggest classifying engine errors (VdcBllMessages) > using these categories: > > 100-199. entity not found > 200-300. missing parameters > 300-399. no resources available > 400-499. can't perform - user error > 500-599. can't perform - server error > 600-699. Unauthorized > 700-x. Forbidden > > this way rest api will be able mapping them to HTTP errors > e.g any error returned by engine CAN_DO_ACTION 100 < x < 200 will be > converted > to HTTP 404 with CAN_DO_ACTION message detail > > - this can be easily done by assigning id to each VdcBllMessages > enum member in range of one of mentioned categories. > > > -- > > Michael Pasternak > RedHat, ENG-Virtualization R&D > > Wouldn't it make more sense for this mapping to happen within the REST API since that is the component that cares about HTTP codes? The engine core should be agnostic to any particular client to ensure its flexibility and separation of concern. -- Michael Pasternak RedHat, ENG-Virtualization R&D From sgordon at redhat.com Tue Nov 22 19:40:37 2011 From: sgordon at redhat.com (Steve Gordon) Date: Tue, 22 Nov 2011 14:40:37 -0500 (EST) Subject: [Engine-devel] Pain Points: Adding NFS Storage In-Reply-To: <2fd8d18f-1f56-4bf6-bfaa-3ccabf715ed4@zmail15.collab.prod.int.phx2.redhat.com> Message-ID: <68d3fc18-a189-462d-976d-468fc8ca17bd@zmail15.collab.prod.int.phx2.redhat.com> Hi guys, By far the most common query that I have seen posed in the IRC channel since the workshop has been around how to successfully add an NFS storage domain. I know there are a host of reasons that this can fail (NFSv4 instead of NFSv3, path is not chown'd 36:36, firewall/server not reachable, bad export configuration) but it's still a pretty obvious pain point and it can be difficult to walk people through troubleshooting all of these possibilities on IRC. I haven't filed a bug around this (yet) but I thought it might be worth trying to come with some ideas for what, if anything, we can change code wise to make this easier for people (and more obvious exactly what is actually wrong when it doesn't work) and then maybe some bugs can be raised from that? I have also created the following page to attempt to collate the requirements for an NFS share to be used by oVirt and how to troubleshoot this issue, please feel free to contribute: http://www.ovirt.org/wiki/Troubleshooting_NFS_Storage_Issues Thanks, Steve From apevec at gmail.com Wed Nov 23 14:10:40 2011 From: apevec at gmail.com (Alan Pevec) Date: Wed, 23 Nov 2011 15:10:40 +0100 Subject: [Engine-devel] Fwd: [Openstack] FOSDEM 2012: Open source Virtualization and Cloud devroom: Call for speakers In-Reply-To: <4ECCF9B9.9050102@openstack.org> References: <4ECCF9B9.9050102@openstack.org> Message-ID: ---------- Forwarded message ---------- From: Thierry Carrez Date: Wed, Nov 23, 2011 at 2:48 PM Subject: [Openstack] FOSDEM 2012: Open source Virtualization and Cloud devroom: Call for speakers To: fosdem at lists.fosdem.org Cc: devrooms at fosdem.org, Lars Kurth , "openstack at lists.launchpad.net" , Renzo Davoli Hello everyone, The devroom named "Open Source Virtualization and Cloud" (OSVC2012) scheduled at FOSDEM on both days, February 4-5 2012, is a meeting opportunity for the developers of all the innovative open source projects around virtualization and cloud computing. OSVC2012 will include: * state-of-the-art presentations of virtualization/cloud projects * workshops to brainstorm and discuss new user requests, new developer ideas, and the possible synergies between projects All information about the devroom will be posted on the wiki at: http://osvc.v2.cs.unibo.it Relevant topics =============== All developers who want to present their ideas and software at OSVC2012 are welcome, provided that: * Their project is related to virtualization or cloud computing * They join the devroom as developers. Developers working for companies, public/private agencies or universities speak for themselves and not for their employer. OSVC2012 is about how to develop more effective virtualizing/cloud solutions (ideas and code, not marketing) * The code for their project must have been released under a free software or open source license (its license must meet the FSF definition of Free Software or the OSI definition of Open Source) Call for Presentations and Workshops ==================================== All submissions must be sent to osvc at cs.unibo.it and include the following information: * Type of submission: Presentation or Workshop * Desired length: "20min presentation + 5min Q&A", "45min presentation + 10min Q&A", or "no preference" * Title * Abstract / Description * Speaker / Moderator bio * Comments: scheduling constraints, remarks... * Links to the project (including its source code) * Related projects Dates: * Deadline for submission: December 18, 2011 * Notification of accepted submissions: January 5, 2012 -- Thierry Carrez Renzo Davoli Lars Kurth _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to ? ? : openstack at lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help ? : https://help.launchpad.net/ListHelp From ranglust at redhat.com Wed Nov 23 14:45:30 2011 From: ranglust at redhat.com (Ronen Angluster) Date: Wed, 23 Nov 2011 16:45:30 +0200 Subject: [Engine-devel] First Nightly build of ovirt-engine & VDSM + Installation Guides Message-ID: <4ECD070A.907@redhat.com> Hello, We've published 2 guides that describe how to install & configure ovirt-engine & VDSM from a nightly build placed on ovirt.org's repository. http://www.ovirt.org/wiki/Installing_ovirt_from_rpm http://www.ovirt.org/wiki/Installing_VDSM_from_rpm Please note the following: 1. At the time, nightly builds are not going to be carried out "nightly", we will publish new versions from time to time until we'll establish a mature and working release process 2. The installation process contains interim steps and will change as we mature the ovirt-engine product's deployment and may change significantly between revisions. Regards, Ronen Angluster From bazulay at redhat.com Wed Nov 23 14:45:54 2011 From: bazulay at redhat.com (Barak Azulay) Date: Wed, 23 Nov 2011 16:45:54 +0200 Subject: [Engine-devel] First Nightly build of ovirt-engine & VDSM + Installation Guides In-Reply-To: <4ECD070A.907@redhat.com> References: <4ECD070A.907@redhat.com> Message-ID: <4ECD0722.2040409@redhat.com> On 11/23/2011 04:45 PM, Ronen Angluster wrote: > Hello, > Great work Ronen. > We've published 2 guides that describe how to install & configure > ovirt-engine & VDSM from a nightly build placed on ovirt.org's repository. > > http://www.ovirt.org/wiki/Installing_ovirt_from_rpm > http://www.ovirt.org/wiki/Installing_VDSM_from_rpm > > Please note the following: > 1. At the time, nightly builds are not going to be carried out "nightly", > we will publish new versions from time to time until we'll establish > a mature and working release process > > 2. The installation process contains interim steps and will change as we > mature the ovirt-engine product's > deployment and may change significantly between revisions. > > > Regards, > Ronen Angluster From marcandre.lureau at gmail.com Wed Nov 23 17:12:25 2011 From: marcandre.lureau at gmail.com (=?ISO-8859-1?Q?Marc=2DAndr=E9_Lureau?=) Date: Wed, 23 Nov 2011 18:12:25 +0100 Subject: [Engine-devel] REST for non-admin account Message-ID: Hi Is there any plan making part of the REST api accessible for non-admin accounts? How can I request and track such a feature? Is ovirt still using bugzilla.redhat.com for bugs? thanks -- Marc-Andr? Lureau From mlipchuk at redhat.com Wed Nov 23 17:31:24 2011 From: mlipchuk at redhat.com (Maor) Date: Wed, 23 Nov 2011 19:31:24 +0200 Subject: [Engine-devel] Quota open issues following oVirt-engine-core meeting Message-ID: <4ECD2DEC.7080904@redhat.com> Hi all. Following the oVirt-engine-core meeting today, and the introduction of the Quota feature, a few open issues were discussed, and are described as follow: 1. Grace limit - We relinquish the grace limit property, since the threshold property should be good enough, for the alert functionality, the administrator needs. 2. Email notifications - The administrator will use the notification service for the Quota threshold event messages, to configure which users, will get notification email on Quota that exceeded the threshold. (We should consider using postponed notifications, to avoid flood). 3, Quota feature scope - Just to clarify, Quota is not designed from hardware point of view, but more from the billing POV. It will only be managed in the ovirt-engine scope. Regards, Maor Lipchuk From dkenigsb at redhat.com Wed Nov 23 16:33:05 2011 From: dkenigsb at redhat.com (Dan Kenigsberg) Date: Wed, 23 Nov 2011 11:33:05 -0500 (EST) Subject: [Engine-devel] First Nightly build of ovirt-engine & VDSM + Installation Guides In-Reply-To: <4ECD070A.907@redhat.com> Message-ID: <66e30c21-e8a5-4ca6-97fb-e45e5fe30473@zmail13.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > Hello, > > We've published 2 guides that describe how to install & configure > ovirt-engine & VDSM from a nightly build placed on ovirt.org's > repository. > > http://www.ovirt.org/wiki/Installing_ovirt_from_rpm > http://www.ovirt.org/wiki/Installing_VDSM_from_rpm I forgot, again, my Wiki password, so I'm complaining here instead of fixing: yum install -y vdsm* is not a good practice. Better do yum install -y vdsm vdsm-cli Dan. From iheim at redhat.com Wed Nov 23 17:34:52 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 23 Nov 2011 19:34:52 +0200 Subject: [Engine-devel] REST for non-admin account In-Reply-To: References: Message-ID: <4ECD2EBC.8020808@redhat.com> On 11/23/2011 07:12 PM, Marc-Andr? Lureau wrote: > Hi > > Is there any plan making part of the REST api accessible for non-admin accounts? very active plans (high on the list), yes. > > How can I request and track such a feature? Is ovirt still using > bugzilla.redhat.com for bugs? bugzilla, under community, ovirt, but still need to migrate RFEs. feel free to open a bug to track this though. > > thanks > From emesika at redhat.com Thu Nov 24 11:09:48 2011 From: emesika at redhat.com (Eli Mesika) Date: Thu, 24 Nov 2011 06:09:48 -0500 (EST) Subject: [Engine-devel] Quota open issues following oVirt-engine-core meeting In-Reply-To: <4ECD2DEC.7080904@redhat.com> Message-ID: <0869b5a0-4c17-48ff-bd34-016c3ae69a8b@zmail13.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > From: "Maor" > To: engine-devel at ovirt.org > Sent: Wednesday, November 23, 2011 7:31:24 PM > Subject: [Engine-devel] Quota open issues following oVirt-engine-core meeting > > Hi all. > Following the oVirt-engine-core meeting today, and the introduction > of > the Quota feature, > a few open issues were discussed, and are described as follow: Can you add those to the Open Issues section in Quota wiki : http://www.ovirt.org/wiki/Features/DetailedQuota#Open_Issues > > 1. Grace limit - We relinquish the grace limit property, since the > threshold property should be good enough, for the alert > functionality, > the administrator needs. > > 2. Email notifications - The administrator will use the notification > service for the Quota threshold event messages, to configure which > users, will get notification email on Quota that exceeded the > threshold. > (We should consider using postponed notifications, to avoid flood). > > 3, Quota feature scope - Just to clarify, Quota is not designed from > hardware point of view, but more from the billing POV. > It will only be managed in the ovirt-engine scope. > > > Regards, > Maor Lipchuk > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From juan.hernandez at redhat.com Thu Nov 24 14:47:36 2011 From: juan.hernandez at redhat.com (Juan Hernandez) Date: Thu, 24 Nov 2011 15:47:36 +0100 Subject: [Engine-devel] The ovirg-mange-domains tool makes super user by design? Message-ID: <4ECE5908.8020307@redhat.com> Hello, I see that the "ovirt-manage-domains" tool updates the "users" and "permissions" table to make the user given in the "-user" option a super user. Is this by design? Shouldn't it just add the domain and leave the "permissions" and "users" table untouched? Thanks in advance, Juan Hernandez From ovedo at redhat.com Thu Nov 24 14:52:21 2011 From: ovedo at redhat.com (Oved Ourfalli) Date: Thu, 24 Nov 2011 09:52:21 -0500 (EST) Subject: [Engine-devel] The ovirg-mange-domains tool makes super user by design? In-Reply-To: <4ECE5908.8020307@redhat.com> Message-ID: ----- Original Message ----- > From: "Juan Hernandez" > To: engine-devel at ovirt.org > Sent: Thursday, November 24, 2011 4:47:36 PM > Subject: [Engine-devel] The ovirg-mange-domains tool makes super user by design? > > Hello, > > I see that the "ovirt-manage-domains" tool updates the "users" and > "permissions" table to make the user given in the "-user" option a > super > user. Is this by design? Shouldn't it just add the domain and leave > the > "permissions" and "users" table untouched? > Yes. It is by design, for the reason that when this user is added, the administrator expects to login with it as well. If we won't add it to the users table, and set permissions to it, then he won't be able to login. I agree that with the introduction of the new admin at internal user, we can think of just setting it in vdc_options, and then if the administrator wants to add users to the new domain he will login with admin at internal, search for users, add them and set the correct permissions. But in order to do that you'll have to make sure the admin at internal user is enabled (i.e., its password is not blank). Otherwise you won't be able to login to the system. > Thanks in advance, > Juan Hernandez > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From mlipchuk at redhat.com Thu Nov 24 15:47:31 2011 From: mlipchuk at redhat.com (Maor) Date: Thu, 24 Nov 2011 17:47:31 +0200 Subject: [Engine-devel] Quota open issues following oVirt-engine-core meeting In-Reply-To: <0869b5a0-4c17-48ff-bd34-016c3ae69a8b@zmail13.collab.prod.int.phx2.redhat.com> References: <0869b5a0-4c17-48ff-bd34-016c3ae69a8b@zmail13.collab.prod.int.phx2.redhat.com> Message-ID: <4ECE6713.7030304@redhat.com> On 11/24/2011 01:09 PM, Eli Mesika wrote: > > > ----- Original Message ----- >> From: "Maor" >> To: engine-devel at ovirt.org >> Sent: Wednesday, November 23, 2011 7:31:24 PM >> Subject: [Engine-devel] Quota open issues following oVirt-engine-core meeting >> >> Hi all. >> Following the oVirt-engine-core meeting today, and the introduction >> of >> the Quota feature, >> a few open issues were discussed, and are described as follow: > > Can you add those to the Open Issues section in Quota wiki : > http://www.ovirt.org/wiki/Features/DetailedQuota#Open_Issues > Done. >> >> 1. Grace limit - We relinquish the grace limit property, since the >> threshold property should be good enough, for the alert >> functionality, >> the administrator needs. >> >> 2. Email notifications - The administrator will use the notification >> service for the Quota threshold event messages, to configure which >> users, will get notification email on Quota that exceeded the >> threshold. >> (We should consider using postponed notifications, to avoid flood). >> >> 3, Quota feature scope - Just to clarify, Quota is not designed from >> hardware point of view, but more from the billing POV. >> It will only be managed in the ovirt-engine scope. >> >> >> Regards, >> Maor Lipchuk >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> From cristi.falcas at gmail.com Fri Nov 25 13:57:37 2011 From: cristi.falcas at gmail.com (Cristian Falcas) Date: Fri, 25 Nov 2011 15:57:37 +0200 Subject: [Engine-devel] Ovirt/VDSM installation Status In-Reply-To: <201111201823.47225.dfediuck@redhat.com> References: <4EC924D4.8040107@redhat.com> <021b01cca79e$1745bf20$45d13d60$@redhat.com> <201111201823.47225.dfediuck@redhat.com> Message-ID: When will tomorrow come? On Sun, Nov 20, 2011 at 18:23, Doron Fediuck wrote: > Good job, guys! > > On Sunday 20 November 2011 18:04:34 Ofer Schreiber wrote: >> Some prof (F16 screen shot) attached. >> >> Ofer. >> >> > -----Original Message----- >> > From: Ronen Angluster [mailto:ranglust at redhat.com] >> > Sent: 20 November 2011 18:04 >> > To: Itamar Heim; Barak Azulay; Ofer Schreiber >> > Cc: Doron Fediuck; engine-devel at ovirt.org; Ayal Baron >> > Subject: Ovirt/VDSM installation Status >> > >> > Hello, >> > >> > We successfully installed Ovirt-engine & VDSM (on the same host!) from >> > the ovirt's repository. >> > >> > The Steps were as following: >> > 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM >> > package containing all the needed information) >> > 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" >> > 3. disable NetworkManager and activating the network service >> > 4. create a bridge interface named "engine" >> > 5. change configuration in /etc/libvirt/qeum.conf >> > /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf >> > 6. configure postgresql for first time installation >> > 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh >> > 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to >> > 'pc-0.14' (known bug in ovirt-engine, patch already submitted) >> > 9. start vdsm & jboss services >> > >> > After those steps were carried, we were able to: >> > 1. add new data center, cluster & host >> > 2. configure storage >> > 3. create and run a new VM >> > >> > a wiki page containing all the necessary information will be delivered >> > tomorrow. >> > >> > Ronen >> >> > > -- > > /d > > ?Funny,? he intoned funereally, ?how just when you think life can't possibly get any worse it suddenly does.? --Douglas Adams, The Hitchhiker's Guide to the Galaxy > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From iheim at redhat.com Fri Nov 25 14:01:24 2011 From: iheim at redhat.com (Itamar Heim) Date: Fri, 25 Nov 2011 09:01:24 -0500 (EST) Subject: [Engine-devel] Ovirt/VDSM installation Status In-Reply-To: References: <4EC924D4.8040107@redhat.com> <021b01cca79e$1745bf20$45d13d60$@redhat.com> <201111201823.47225.dfediuck@redhat.com> Message-ID: <966f01ccab7a$5a8ad6f0$0fa084d0$@com> > -----Original Message----- > From: engine-devel-bounces at ovirt.org [mailto:engine-devel-bounces at ovirt.org] On Behalf Of Cristian Falcas > Sent: Friday, November 25, 2011 15:58 PM > To: Doron Fediuck > Cc: engine-devel at ovirt.org; Ayal Baron > Subject: Re: [Engine-devel] Ovirt/VDSM installation Status > > When will tomorrow come? I think Ronen sent these links to the list already: http://www.ovirt.org/wiki/Installing_ovirt-engine_from_rpm http://www.ovirt.org/wiki/Installing_VDSM_from_rpm > > On Sun, Nov 20, 2011 at 18:23, Doron Fediuck wrote: > > Good job, guys! > > > > On Sunday 20 November 2011 18:04:34 Ofer Schreiber wrote: > >> Some prof (F16 screen shot) attached. > >> > >> Ofer. > >> > >> > -----Original Message----- > >> > From: Ronen Angluster [mailto:ranglust at redhat.com] > >> > Sent: 20 November 2011 18:04 > >> > To: Itamar Heim; Barak Azulay; Ofer Schreiber > >> > Cc: Doron Fediuck; engine-devel at ovirt.org; Ayal Baron > >> > Subject: Ovirt/VDSM installation Status > >> > > >> > Hello, > >> > > >> > We successfully installed Ovirt-engine & VDSM (on the same host!) from > >> > the ovirt's repository. > >> > > >> > The Steps were as following: > >> > 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM > >> > package containing all the needed information) > >> > 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" > >> > 3. disable NetworkManager and activating the network service > >> > 4. create a bridge interface named "engine" > >> > 5. change configuration in /etc/libvirt/qeum.conf > >> > /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf > >> > 6. configure postgresql for first time installation > >> > 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh > >> > 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to > >> > 'pc-0.14' (known bug in ovirt-engine, patch already submitted) > >> > 9. start vdsm & jboss services > >> > > >> > After those steps were carried, we were able to: > >> > 1. add new data center, cluster & host > >> > 2. configure storage > >> > 3. create and run a new VM > >> > > >> > a wiki page containing all the necessary information will be delivered > >> > tomorrow. > >> > > >> > Ronen > >> > >> > > > > -- > > > > /d > > > > "Funny," he intoned funereally, "how just when you think life can't possibly get any worse it suddenly > does." --Douglas Adams, The Hitchhiker's Guide to the Galaxy > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From cristi.falcas at gmail.com Fri Nov 25 16:52:52 2011 From: cristi.falcas at gmail.com (Cristian Falcas) Date: Fri, 25 Nov 2011 18:52:52 +0200 Subject: [Engine-devel] Ovirt/VDSM installation Status In-Reply-To: <966f01ccab7a$5a8ad6f0$0fa084d0$@com> References: <4EC924D4.8040107@redhat.com> <021b01cca79e$1745bf20$45d13d60$@redhat.com> <201111201823.47225.dfediuck@redhat.com> <966f01ccab7a$5a8ad6f0$0fa084d0$@com> Message-ID: thank you On Fri, Nov 25, 2011 at 16:01, Itamar Heim wrote: > > >> -----Original Message----- >> From: engine-devel-bounces at ovirt.org > [mailto:engine-devel-bounces at ovirt.org] On Behalf Of Cristian Falcas >> Sent: Friday, November 25, 2011 15:58 PM >> To: Doron Fediuck >> Cc: engine-devel at ovirt.org; Ayal Baron >> Subject: Re: [Engine-devel] Ovirt/VDSM installation Status >> >> When will tomorrow come? > > I think Ronen sent these links to the list already: > http://www.ovirt.org/wiki/Installing_ovirt-engine_from_rpm > http://www.ovirt.org/wiki/Installing_VDSM_from_rpm > > >> >> On Sun, Nov 20, 2011 at 18:23, Doron Fediuck > wrote: >> > Good job, guys! >> > >> > On Sunday 20 November 2011 18:04:34 Ofer Schreiber wrote: >> >> Some prof (F16 screen shot) attached. >> >> >> >> Ofer. >> >> >> >> > -----Original Message----- >> >> > From: Ronen Angluster [mailto:ranglust at redhat.com] >> >> > Sent: 20 November 2011 18:04 >> >> > To: Itamar Heim; Barak Azulay; Ofer Schreiber >> >> > Cc: Doron Fediuck; engine-devel at ovirt.org; Ayal Baron >> >> > Subject: Ovirt/VDSM installation Status >> >> > >> >> > Hello, >> >> > >> >> > We successfully installed Ovirt-engine & VDSM (on the same host!) > from >> >> > the ovirt's repository. >> >> > >> >> > The Steps were as following: >> >> > 1. add a repository to /etc/yum.repo.d (will be replaced by a RPM >> >> > package containing all the needed information) >> >> > 2. execute: "yum install -y ovirt-engine; yum install -y vdsm*" >> >> > 3. disable NetworkManager and activating the network service >> >> > 4. create a bridge interface named "engine" >> >> > 5. change configuration in /etc/libvirt/qeum.conf >> >> > /etc/libvirt/libvirtd.conf & /etc/vdsm/vdsm.conf >> >> > 6. configure postgresql for first time installation >> >> > 7. execute /usr/share/ovirt-engine/dbscripts/create_db_devel.sh >> >> > 8. change vdc_options option 'EmulatedMachine' from 'rhel6.2.0' to >> >> > 'pc-0.14' (known bug in ovirt-engine, patch already submitted) >> >> > 9. start vdsm & jboss services >> >> > >> >> > After those steps were carried, we were able to: >> >> > 1. add new data center, cluster & host >> >> > 2. configure storage >> >> > 3. create and run a new VM >> >> > >> >> > a wiki page containing all the necessary information will be > delivered >> >> > tomorrow. >> >> > >> >> > Ronen >> >> >> >> >> > >> > -- >> > >> > /d >> > >> > "Funny," he intoned funereally, "how just when you think life can't > possibly get any worse it suddenly >> does." --Douglas Adams, The Hitchhiker's Guide to the Galaxy >> > _______________________________________________ >> > Engine-devel mailing list >> > Engine-devel at ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/engine-devel >> > >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel > From gjansen at redhat.com Sat Nov 26 17:49:33 2011 From: gjansen at redhat.com (Geert Jansen) Date: Sat, 26 Nov 2011 18:49:33 +0100 Subject: [Engine-devel] Feedback on ovirt-engine-sdk Message-ID: <4ED126AD.7000804@redhat.com> Hi Michael, i just wrote a small program with the new Python SDK. I found it very easy to use, so great work. I do have a few points of feedback: * Probably we want all Exceptions derived from a single base class. That way, the user can intercept any ovirtsdk with a single try/except clause. Currently that's not possible, as all your exceptions derive directly from Exception. * I think the .get() method should be id-based not name based. Using "id" as a primary key has a number of advantages over name: - It is immutable, which makes e.g. renaming a lot easier. - It allows you to retrieve an object without a search query or a filter. - It matches how the API works as well. Also currently i think it's not possible to retrieve a resource by its ID, unless you use the id= keyword argument to list() which retrieves the whole collection and then filters out the matching id. * You may want to introduce some non-entity related operations, like ping(), and connect(). These allow you to better control when you connect, and allow an program to test if the connection details work. That helps client except the right exceptions at the right time, and display more relevant error messages. * There's two Python objects involved for each API object. For a VM for example, you have params.VM and brokers.VM. This distinction is made visible to the user, for example: template = api.templates.get('Blank') vm = params.VM() vm.name = 'foo' vm.template = template # ERROR The last line need to be: vm.template = params.Template(id=template.id) In my version of the Python API i used mix-in classes to add the OO behavior to the generated classes. It was easy, because PyXB had support for that. This gave me the single class behavior. I am not sure if GenerateDS generated code allows that. I'm pretty sure it could be done though, but you may need to do some Python magic. * Nitpick: should this just be called "ovirt" instead of "ovirtsdk"? Overall great work though! Regards, Geert From mpastern at redhat.com Sun Nov 27 09:16:26 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Sun, 27 Nov 2011 11:16:26 +0200 Subject: [Engine-devel] Feedback on ovirt-engine-sdk In-Reply-To: <4ED126AD.7000804@redhat.com> References: <4ED126AD.7000804@redhat.com> Message-ID: <4ED1FFEA.20003@redhat.com> Hi Geert, On 11/26/2011 07:49 PM, Geert Jansen wrote: > Hi Michael, > > i just wrote a small program with the new Python SDK. I found it very easy to use, so great work. I do have a few points of feedback: > > * Probably we want all Exceptions derived from a single base class. That way, the user can intercept any ovirtsdk with a single try/except clause. Currently that's not > possible, as all your exceptions derive directly from Exception. sure, in my TODO list. > > * I think the .get() method should be id-based not name based. Using "id" as a primary key has a number of advantages over name: > > - It is immutable, which makes e.g. renaming a lot easier. > - It allows you to retrieve an object without a search query or a filter. > - It matches how the API works as well. "id" is supported, you can do api.collection.get(id=xxx), only as you mentioned "id" is not primary, but secondary key (i thought that searching by name will be easier), but i can change it in a second if this is an issue. > > Also currently i think it's not possible to retrieve a resource by its ID, unless you use the id= keyword argument to list() which retrieves the whole collection and then > filters out the matching id. nope, it's supported - i even added it to examples.py [1] -> vm5 = api.vms.get(id='02f0f4a4-9738-4731-83c4-293f3f734782') [1] will add it to get method .get(id, name) as you suggested. > > * You may want to introduce some non-entity related operations, like ping(), and connect(). These allow you to better control when you connect, and allow an program to > test if the connection details work. That helps client except the right exceptions at the right time, and display more relevant error messages. - no need for explicit connect() as its done implicitly on proxy construction - no need for ping() as any attempt to use disconnected (and not recoverable) proxy, will cause ConnectionError, so user should only /Trap/ this exception for disaster_recovery > > * There's two Python objects involved for each API object. For a VM for example, you have params.VM and brokers.VM. This distinction is made visible to the user, for example: > > template = api.templates.get('Blank') > > vm = params.VM() > vm.name = 'foo' > vm.template = template # ERROR yep, you right doing this is in my TODO list as well (had to switch to CLI, so i do have few leftovers[1] - this is one of them) [1] they all have simple workarounds (like this one as described below) > > The last line need to be: > > vm.template = params.Template(id=template.id) for now (as workaround you can pass 'superclass', e.g vm.template = template.superclass), but i suggest you just wait for another day till I'll support this. > > In my version of the Python API i used mix-in classes to add the OO behavior to the generated classes. It was easy, because PyXB had support for that. This gave me the > single class behavior. I am not sure if GenerateDS generated code allows that. I'm pretty sure it could be done though, but you may need to do some Python magic. no need, i do this with regular inheritance pattern > > * Nitpick: should this just be called "ovirt" instead of "ovirtsdk"? initially it was, but since both cli & sdk would operate in same namespace, i wanted to avoid future collisions and renamed them to ovirtsdk/ovirtcli. > > Overall great work though! 10nx a lot mate. > > Regards, > Geert > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel -- Michael Pasternak RedHat, ENG-Virtualization R&D From iheim at redhat.com Sun Nov 27 09:41:47 2011 From: iheim at redhat.com (Itamar Heim) Date: Sun, 27 Nov 2011 11:41:47 +0200 Subject: [Engine-devel] [rhevm-api] Event notifications in RHEVM 3 In-Reply-To: References: <4ECD6D07.30405@redhat.com> , <4ECDE9A1.1040801@redhat.com> <284C7578-D3C9-45AF-9849-359360199AC0@virtustream.com> <4ECE3629.6060202@redhat.com> <4ECE4328.3080107@redhat.com> <4ECE4CD9.3030109@redhat.com> Message-ID: <4ED205DB.3090200@redhat.com> On 11/25/2011 02:48 PM, David Curylo wrote: > In our use case, we are trying to get real time notifications of events into our own application tier, so there would only be one or two of our application servers connected at a time. It need not be REST or HTTP - a persistent TCP connection will work just as well for us, the importance is that we are notified as quickly as possible. We have a similar need for real time notifications throughout our system and our architecture can be summarized as follows: > > Application server(s) --> AMQP topic exchange --> AMQP queue topic binding --> Web server(s) > > I will share more of our usage scenarios and a little about our architecture as it pertains to the discussion: > > When our application tier is notified of events, it performs other actions. For example, for Red Hat certification, we track the power off and power on events so we can provide a report of OS usage - polling every minute or so would work fine here. We also need to notify server monitoring systems that a system was powered off so they can stop monitoring rather than alerting of an outage - in this case, we need to be notified ASAP. Additionally, we need to notify any connected clients that a system has changed state (any event). Those clients are connected to web servers, which we notify by a message queuing infrastructure. All events are published to an AMQP topic exchange, to which web servers (and other systems) may subscribe based on their connected client's needs. The transport and scalability requirements no longer need to be handled by our application tier. > > The web servers, in the end, have to support a lot of client scenarios that include polling, long polling, sending HTTP requests, maintaining a TCP channel for streaming events, etc. We can easily scale up web servers, and we need to provide transports that are suitable for whatever our clients may need. > > I hope this helps explain our usage scenarios and suggest an additional technology to consider. (moving the discussion to the new upstream mailing list) do you care about all events, or specific ones? the event notifier is about sending registered events per user. maybe extend it to send the amqp events directly / extend the list of events it cares about? > > -----Original Message----- > From: Yaniv Kaul [mailto:ykaul at redhat.com] > Sent: Thursday, November 24, 2011 8:56 AM > To: Geert Jansen > Cc: Itamar Heim; rhevm-api at lists.fedorahosted.org; David Curylo > Subject: Re: [rhevm-api] Event notifications in RHEVM 3 > > On 11/24/2011 03:14 PM, Geert Jansen wrote: >> >> On 11/24/2011 01:18 PM, Itamar Heim wrote: >>> On 11/24/2011 01:37 PM, David Curylo wrote: >>>> I don't know the capabilities of this daemon. Can it do other notifications besides email, such as execute curl to send a request to my system? I would like to get notifications as close to real time as possible. >>> I think it's every 5 minutes right now, but can be configured (in 3.0). >> I don't believe though, that out notification service suits this use >> case. The use case is to get almost real-time notifications via an API >> when a change has been made. >> >> In my view, the best way to do this in HTTP/REST, is so-called long >> polling. It's not perfect, and not terribly scalable (i takes up one >> TCP connection, and in the simplest implementation, one app server slot). >> But on the other side, i do not see a need for 100s of clients polling >> a single RHEV systems. Typically, a customer may have 1-2 ISV packages >> on top of RHEV that need events. In my view, 10s of clients should be >> easily doable with long polling. > > - A single persistent TCP connection is "cheaper" to everyone than open-request-response-close-open-request-response-close .... > - This is widely used over HTTP with "100 continue" messages that never actually complete with a 200. It's neat and takes very little bandwidth, actually. > Y. > >> >> If we want to scale higher than this, probably a different >> architecture like e.g. a pub/sub architecture using a protocol >> optimized for binary messaging would make sense. But again, for now, i >> have not seen this need so for 3.1 i think we need to focus on the >> simplest solution that will satisfy 95% of the requirements. >> >> Regards, >> Geert >> _______________________________________________ >> rhevm-api mailing list >> rhevm-api at lists.fedorahosted.org >> https://fedorahosted.org/mailman/listinfo/rhevm-api > From ykaul at redhat.com Sun Nov 27 10:18:47 2011 From: ykaul at redhat.com (Yaniv Kaul) Date: Sun, 27 Nov 2011 12:18:47 +0200 Subject: [Engine-devel] Feedback on ovirt-engine-sdk In-Reply-To: <4ED1FFEA.20003@redhat.com> References: <4ED126AD.7000804@redhat.com> <4ED1FFEA.20003@redhat.com> Message-ID: <4ED20E87.6020005@redhat.com> On 11/27/2011 11:16 AM, Michael Pasternak wrote: > Hi Geert, > > On 11/26/2011 07:49 PM, Geert Jansen wrote: >> Hi Michael, >> >> i just wrote a small program with the new Python SDK. I found it very easy to use, so great work. I do have a few points of feedback: >> >> * Probably we want all Exceptions derived from a single base class. That way, the user can intercept any ovirtsdk with a single try/except clause. Currently that's not >> possible, as all your exceptions derive directly from Exception. > sure, in my TODO list. Would be great if you could publish the TODO list on the ovirt Wiki. It would encourage others to work on those items as well. Y. > >> * I think the .get() method should be id-based not name based. Using "id" as a primary key has a number of advantages over name: >> >> - It is immutable, which makes e.g. renaming a lot easier. >> - It allows you to retrieve an object without a search query or a filter. >> - It matches how the API works as well. > "id" is supported, you can do api.collection.get(id=xxx), only as you mentioned > "id" is not primary, but secondary key (i thought that searching by name will be easier), > but i can change it in a second if this is an issue. > >> Also currently i think it's not possible to retrieve a resource by its ID, unless you use the id= keyword argument to list() which retrieves the whole collection and then >> filters out the matching id. > nope, it's supported - i even added it to examples.py [1] > -> vm5 = api.vms.get(id='02f0f4a4-9738-4731-83c4-293f3f734782') > > [1] will add it to get method .get(id, name) as you suggested. > >> * You may want to introduce some non-entity related operations, like ping(), and connect(). These allow you to better control when you connect, and allow an program to >> test if the connection details work. That helps client except the right exceptions at the right time, and display more relevant error messages. > - no need for explicit connect() as its done implicitly on proxy construction > - no need for ping() as any attempt to use disconnected (and not recoverable) proxy, will cause ConnectionError, > so user should only /Trap/ this exception for disaster_recovery > >> * There's two Python objects involved for each API object. For a VM for example, you have params.VM and brokers.VM. This distinction is made visible to the user, for example: >> >> template = api.templates.get('Blank') >> >> vm = params.VM() >> vm.name = 'foo' >> vm.template = template # ERROR > yep, you right doing this is in my TODO list as well > (had to switch to CLI, so i do have few leftovers[1] - this is one of them) > > [1] they all have simple workarounds (like this one as described below) > >> The last line need to be: >> >> vm.template = params.Template(id=template.id) > for now (as workaround you can pass 'superclass', e.g vm.template = template.superclass), > but i suggest you just wait for another day till I'll support this. > >> In my version of the Python API i used mix-in classes to add the OO behavior to the generated classes. It was easy, because PyXB had support for that. This gave me the >> single class behavior. I am not sure if GenerateDS generated code allows that. I'm pretty sure it could be done though, but you may need to do some Python magic. > no need, i do this with regular inheritance pattern > >> * Nitpick: should this just be called "ovirt" instead of "ovirtsdk"? > initially it was, but since both cli& sdk would operate in same namespace, > i wanted to avoid future collisions and renamed them to ovirtsdk/ovirtcli. > >> Overall great work though! > 10nx a lot mate. > >> Regards, >> Geert >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel > From mpastern at redhat.com Sun Nov 27 10:39:30 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Sun, 27 Nov 2011 12:39:30 +0200 Subject: [Engine-devel] Feedback on ovirt-engine-sdk In-Reply-To: <4ED20E87.6020005@redhat.com> References: <4ED126AD.7000804@redhat.com> <4ED1FFEA.20003@redhat.com> <4ED20E87.6020005@redhat.com> Message-ID: <4ED21362.2050905@redhat.com> On 11/27/2011 12:18 PM, Yaniv Kaul wrote: > Would be great if you could publish the TODO list on the ovirt Wiki. > It would encourage others to work on those items as well. > Y. absolutely, will do. thanks. -- Michael Pasternak RedHat, ENG-Virtualization R&D From mkenneth at redhat.com Sun Nov 27 12:17:29 2011 From: mkenneth at redhat.com (Miki Kenneth) Date: Sun, 27 Nov 2011 07:17:29 -0500 (EST) Subject: [Engine-devel] Quota feature description In-Reply-To: <4EC54F76.8000909@redhat.com> Message-ID: I think that capping the number of snapshots is a very intutive idea, and it might be easier for cloud provider use case. However, I don't think we can omit the total space option, but having a cap on the number of snapshots in addition, sounds fine. On another note, looking at some other storage vendors I can say that in most cases the following parameters apply: - Hard Limit - when exceeding it the action is denied. - Soft limit - when exceeding it the User get a warning. (optional) - Grace - on the Hard limit, that in some cases is restricted by time as well So I do think for completeness we should define/design on these parameters. Miki ----- Original Message ----- > From: "Maor" > To: "Saggi Mizrahi" > Cc: "Oved Ourfalli" , engine-devel at ovirt.org > Sent: Thursday, November 17, 2011 8:16:22 PM > Subject: Re: [Engine-devel] Quota feature description > > I updated the detailedQuota wiki, to sharpen that the Quota is > managed > from the engine-core perspective. > > Your suggestion about the number of snapshots per image is quite > interesting, I think a PM should give an opinion about it. > Although I'm not sure the limitation, should be in the Quota scope, > maybe it is more accurate to set it as a limit in the VM scope. or > the > image scope instead. > > > On 11/17/2011 05:41 PM, Saggi Mizrahi wrote: > > On Thu 17 Nov 2011 03:29:48 PM IST, Maor wrote: > >> Hi Saggi, thanks for the comments, please see my comments in line > >> > >> On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: > >>> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: > >>>> Hello all, > >>>> > >>>> The Quota feature description is published under the following > >>>> links: > >>>> http://www.ovirt.org/wiki/Features/DetailedQuota > >>>> http://www.ovirt.org/wiki/Features/Quota > >>>> Notice that screens of UI mockups should be updated. > >>>> > >>>> Please feel free, to share your comments about it. > >>>> > >>>> Thank you, > >>>> Maor > >>>> > >>>> _______________________________________________ > >>>> Engine-devel mailing list > >>>> Engine-devel at ovirt.org > >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>> > >>> I can't see how the host is supposed to enforce and handle it. > >>> Pause the > >>> VM? Crash it? raise ENOMEM\ENOSPC in the guest? > >> The enforcement and handling, should be from the engine scope and > >> not > >> from the Host perspective. > >> Actually the Host, should not be aware of Quota at all. > >>> Also what about cases of KSM\QCow2, disk\memory overcommit. > >> on QCOW issue, the active disk should consume the full potential > >> space > >> from the Quota, since we are not sure how much space will be in > >> use, > >> slthough the snapshot disk, will be updated to consume only its > >> real > >> size from the Quota. > >> > >> you can check the Enforcement section : > >> "When dealing with QCOW disks (which is not pre-allocated, like > >> templates or stateless VM) the Quota should consume the total > >> maximum > >> size of the disk, since it is the potential size that can be > >> used." > >> > >> for overcommit issue, please see CRUD section in the WIKI: > >> "...However, users will not be able to exceed the Quota > >> limitations > >> again after the resources are released." > >> > >>> Disk Templates. > >>> Storage for hibernation disk. > >>> Temporary and shared disks. > >> same logic as above (Enforcement section) > >>> Shared disks between VMs owned by different users. > >> Please see Dependencies / Related Features and Projects: > >> "When handling plug/unplug disks or attach/detach disks, the > >> entity will > >> still consume resources from its configured original Quota it was > >> created on. " > >> > >> Which means the disk should consume from the same Quota all the > >> time > >> (not dependent on the users that use it). > >>> Backup snapshots (should they count in the quota? They are > >>> transient) > >> When ever a volume is created whether it is snapshot, backup > >> snapshot, > >> stateless disk, or any QCOW implementation, the enforcement should > >> the > >> the same as described above (see Enforcement section) > >>> > >>> I also don't see how vcpu limiting is any good? I don't even know > >>> what > >>> it means. What happens in migration between hosts with different > >>> amount > >>> of physical CPUs? > >> The "atomic" section that Quota is handling in the run time scope > >> is > >> cluster. > >> Actually for the user migration will be transparent since it is > >> consumed > >> from the same Quota, the only validation the VM should encounter > >> will be > >> the same as before in the Host perspective. > >> > >>> I also don't think CPU limiting is even a thing to be managed by > >>> a > >>> quota. There is no reason not to use 100% of the CPU if you are > >>> the only > >>> VM active. CPU scheduling should use a priority model and not a > >>> quota > >>> IMHO. > >> Again, the Quota should be managed from the engine level, and > >> should not > >> be reflected in the Host implementation. > >> Try to look at it, as an abstract management mechanism for taking > >> notes > >> and managing resource consumes for the Administrator. > >> > >> A priority model is an interesting thought. > >> Now it can be supported, by using different grace percentage from > >> one > >> Quota to another, or maybe create different Quota for Different > >> type of > >> users. > > > > So I understand there is a pure disconnect between host lever > > quotas and > > policies and this quota feature. > > Speaking with Livnat and Maor I see this is a good use case for an > > ovirt > > user to allow simple quotas to clients. > > > > It just feels like a feature that should be implemented as part of > > a > > grater policy structure validating user actions and not as a strict > > quota. > > > > > > I think snapshots and images should have different quotas where > > images > > are capped by size and snapshots are capped by number. > > > > Think of it from a hosting provider perspective. I want each client > > to > > get a maximum of 30GB of storage. > > I can limit them to 30GB. And let them calculated that if they made > > a > > 16GB worth of VM they can never snapshot it. On the other hand I > > can > > chose to limit them to 10GB and allow up to 2 snapshots to each > > clients. > > This way I know each client will take up to 30 GB and there will be > > no > > complaints when the user created a VM and now he can't snapshot it. > > Snapshotting can also be capped on the image level instead of the > > user > > level (3 snapshots per image). I tend to go towards the latter > > because > > it'll keep multiple disk VM snapshots from failing due to depleted > > quota. > > > > This will also make complaints about the pessimistic storage > > estimation > > invalid. > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From mlipchuk at redhat.com Sun Nov 27 14:16:32 2011 From: mlipchuk at redhat.com (Maor) Date: Sun, 27 Nov 2011 16:16:32 +0200 Subject: [Engine-devel] Quota feature description In-Reply-To: References: Message-ID: <4ED24640.8070001@redhat.com> On 11/27/2011 02:17 PM, Miki Kenneth wrote: > I think that capping the number of snapshots is a very intutive idea, and it might be easier for cloud provider use case. However, I don't think we can omit the total space option, but having a cap on the number of snapshots in addition, sounds fine. sounds good, but it raises a few issues: First, if we already thinking of limiting the number of business entities, like snapshot, why not also limit the number of other entities like disks. because the limitations of snapshot number, actually reflects on all the snapshot for all the VMs created on the Quota. What happened if someone wants to use on his VM, more then the snapshot configured. The administrator will have to change the configuration of the Quota for all the VMs or reallocate the disk on other Quota that enables it (If there is one). Also I hope we don't give the user too many configuration parameters without making the feature to be too complex to use. maybe it is a requirement, that we should wait for feedbacks on it, see if this case is usable enough... > > On another note, looking at some other storage vendors I can say that in most cases the following parameters apply: > - Hard Limit - when exceeding it the action is denied. > - Soft limit - when exceeding it the User get a warning. (optional) > - Grace - on the Hard limit, that in some cases is restricted by time as well > So I do think for completeness we should define/design on these parameters. Agreed. > > Miki > > > ----- Original Message ----- >> From: "Maor" >> To: "Saggi Mizrahi" >> Cc: "Oved Ourfalli" , engine-devel at ovirt.org >> Sent: Thursday, November 17, 2011 8:16:22 PM >> Subject: Re: [Engine-devel] Quota feature description >> >> I updated the detailedQuota wiki, to sharpen that the Quota is >> managed >> from the engine-core perspective. >> >> Your suggestion about the number of snapshots per image is quite >> interesting, I think a PM should give an opinion about it. >> Although I'm not sure the limitation, should be in the Quota scope, >> maybe it is more accurate to set it as a limit in the VM scope. or >> the >> image scope instead. >> >> >> On 11/17/2011 05:41 PM, Saggi Mizrahi wrote: >>> On Thu 17 Nov 2011 03:29:48 PM IST, Maor wrote: >>>> Hi Saggi, thanks for the comments, please see my comments in line >>>> >>>> On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: >>>>> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: >>>>>> Hello all, >>>>>> >>>>>> The Quota feature description is published under the following >>>>>> links: >>>>>> http://www.ovirt.org/wiki/Features/DetailedQuota >>>>>> http://www.ovirt.org/wiki/Features/Quota >>>>>> Notice that screens of UI mockups should be updated. >>>>>> >>>>>> Please feel free, to share your comments about it. >>>>>> >>>>>> Thank you, >>>>>> Maor >>>>>> >>>>>> _______________________________________________ >>>>>> Engine-devel mailing list >>>>>> Engine-devel at ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>>> >>>>> I can't see how the host is supposed to enforce and handle it. >>>>> Pause the >>>>> VM? Crash it? raise ENOMEM\ENOSPC in the guest? >>>> The enforcement and handling, should be from the engine scope and >>>> not >>>> from the Host perspective. >>>> Actually the Host, should not be aware of Quota at all. >>>>> Also what about cases of KSM\QCow2, disk\memory overcommit. >>>> on QCOW issue, the active disk should consume the full potential >>>> space >>>> from the Quota, since we are not sure how much space will be in >>>> use, >>>> slthough the snapshot disk, will be updated to consume only its >>>> real >>>> size from the Quota. >>>> >>>> you can check the Enforcement section : >>>> "When dealing with QCOW disks (which is not pre-allocated, like >>>> templates or stateless VM) the Quota should consume the total >>>> maximum >>>> size of the disk, since it is the potential size that can be >>>> used." >>>> >>>> for overcommit issue, please see CRUD section in the WIKI: >>>> "...However, users will not be able to exceed the Quota >>>> limitations >>>> again after the resources are released." >>>> >>>>> Disk Templates. >>>>> Storage for hibernation disk. >>>>> Temporary and shared disks. >>>> same logic as above (Enforcement section) >>>>> Shared disks between VMs owned by different users. >>>> Please see Dependencies / Related Features and Projects: >>>> "When handling plug/unplug disks or attach/detach disks, the >>>> entity will >>>> still consume resources from its configured original Quota it was >>>> created on. " >>>> >>>> Which means the disk should consume from the same Quota all the >>>> time >>>> (not dependent on the users that use it). >>>>> Backup snapshots (should they count in the quota? They are >>>>> transient) >>>> When ever a volume is created whether it is snapshot, backup >>>> snapshot, >>>> stateless disk, or any QCOW implementation, the enforcement should >>>> the >>>> the same as described above (see Enforcement section) >>>>> >>>>> I also don't see how vcpu limiting is any good? I don't even know >>>>> what >>>>> it means. What happens in migration between hosts with different >>>>> amount >>>>> of physical CPUs? >>>> The "atomic" section that Quota is handling in the run time scope >>>> is >>>> cluster. >>>> Actually for the user migration will be transparent since it is >>>> consumed >>>> from the same Quota, the only validation the VM should encounter >>>> will be >>>> the same as before in the Host perspective. >>>> >>>>> I also don't think CPU limiting is even a thing to be managed by >>>>> a >>>>> quota. There is no reason not to use 100% of the CPU if you are >>>>> the only >>>>> VM active. CPU scheduling should use a priority model and not a >>>>> quota >>>>> IMHO. >>>> Again, the Quota should be managed from the engine level, and >>>> should not >>>> be reflected in the Host implementation. >>>> Try to look at it, as an abstract management mechanism for taking >>>> notes >>>> and managing resource consumes for the Administrator. >>>> >>>> A priority model is an interesting thought. >>>> Now it can be supported, by using different grace percentage from >>>> one >>>> Quota to another, or maybe create different Quota for Different >>>> type of >>>> users. >>> >>> So I understand there is a pure disconnect between host lever >>> quotas and >>> policies and this quota feature. >>> Speaking with Livnat and Maor I see this is a good use case for an >>> ovirt >>> user to allow simple quotas to clients. >>> >>> It just feels like a feature that should be implemented as part of >>> a >>> grater policy structure validating user actions and not as a strict >>> quota. >>> >>> >>> I think snapshots and images should have different quotas where >>> images >>> are capped by size and snapshots are capped by number. >>> >>> Think of it from a hosting provider perspective. I want each client >>> to >>> get a maximum of 30GB of storage. >>> I can limit them to 30GB. And let them calculated that if they made >>> a >>> 16GB worth of VM they can never snapshot it. On the other hand I >>> can >>> chose to limit them to 10GB and allow up to 2 snapshots to each >>> clients. >>> This way I know each client will take up to 30 GB and there will be >>> no >>> complaints when the user created a VM and now he can't snapshot it. >>> Snapshotting can also be capped on the image level instead of the >>> user >>> level (3 snapshots per image). I tend to go towards the latter >>> because >>> it'll keep multiple disk VM snapshots from failing due to depleted >>> quota. >>> >>> This will also make complaints about the pessimistic storage >>> estimation >>> invalid. >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> From mkenneth at redhat.com Sun Nov 27 15:40:13 2011 From: mkenneth at redhat.com (Miki Kenneth) Date: Sun, 27 Nov 2011 10:40:13 -0500 (EST) Subject: [Engine-devel] Quota feature description In-Reply-To: <4ED24640.8070001@redhat.com> Message-ID: <00129649-0eb9-4ed7-94b7-a4802dd0e90c@mkenneth.csb> ----- Original Message ----- > From: "Maor" > To: "Miki Kenneth" > Cc: "Oved Ourfalli" , engine-devel at ovirt.org, "Saggi Mizrahi" > Sent: Sunday, November 27, 2011 4:16:32 PM > Subject: Re: [Engine-devel] Quota feature description > > On 11/27/2011 02:17 PM, Miki Kenneth wrote: > > I think that capping the number of snapshots is a very intutive > > idea, and it might be easier for cloud provider use case. However, > > I don't think we can omit the total space option, but having a cap > > on the number of snapshots in addition, sounds fine. > sounds good, but it raises a few issues: > First, if we already thinking of limiting the number of business > entities, like snapshot, why not also limit the number of other > entities > like disks. Don't see a reason for that. > > because the limitations of snapshot number, actually reflects on all > the > snapshot for all the VMs created on the Quota. > What happened if someone wants to use on his VM, more then the > snapshot > configured. > The administrator will have to change the configuration of the Quota > for > all the VMs or reallocate the disk on other Quota that enables it (If > there is one). I see that as another of "global" quota. I would not go as far as specifying a number per SD. > > Also I hope we don't give the user too many configuration parameters > without making the feature to be too complex to use. > maybe it is a requirement, that we should wait for feedbacks on it, > see > if this case is usable enough... As a role, I'm all for it to make it simple and get FB. I do think it is a nice idea and a nice to have - can be pushed to > v2. > > > > > On another note, looking at some other storage vendors I can say > > that in most cases the following parameters apply: > > - Hard Limit - when exceeding it the action is denied. > > - Soft limit - when exceeding it the User get a warning. (optional) > > - Grace - on the Hard limit, that in some cases is restricted by > > time as well > > So I do think for completeness we should define/design on these > > parameters. > Agreed. > > > > Miki > > > > > > ----- Original Message ----- > >> From: "Maor" > >> To: "Saggi Mizrahi" > >> Cc: "Oved Ourfalli" , engine-devel at ovirt.org > >> Sent: Thursday, November 17, 2011 8:16:22 PM > >> Subject: Re: [Engine-devel] Quota feature description > >> > >> I updated the detailedQuota wiki, to sharpen that the Quota is > >> managed > >> from the engine-core perspective. > >> > >> Your suggestion about the number of snapshots per image is quite > >> interesting, I think a PM should give an opinion about it. > >> Although I'm not sure the limitation, should be in the Quota > >> scope, > >> maybe it is more accurate to set it as a limit in the VM scope. or > >> the > >> image scope instead. > >> > >> > >> On 11/17/2011 05:41 PM, Saggi Mizrahi wrote: > >>> On Thu 17 Nov 2011 03:29:48 PM IST, Maor wrote: > >>>> Hi Saggi, thanks for the comments, please see my comments in > >>>> line > >>>> > >>>> On 11/17/2011 02:36 PM, Saggi Mizrahi wrote: > >>>>> On Wed 16 Nov 2011 02:48:40 PM IST, Maor wrote: > >>>>>> Hello all, > >>>>>> > >>>>>> The Quota feature description is published under the following > >>>>>> links: > >>>>>> http://www.ovirt.org/wiki/Features/DetailedQuota > >>>>>> http://www.ovirt.org/wiki/Features/Quota > >>>>>> Notice that screens of UI mockups should be updated. > >>>>>> > >>>>>> Please feel free, to share your comments about it. > >>>>>> > >>>>>> Thank you, > >>>>>> Maor > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Engine-devel mailing list > >>>>>> Engine-devel at ovirt.org > >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>>>> > >>>>> I can't see how the host is supposed to enforce and handle it. > >>>>> Pause the > >>>>> VM? Crash it? raise ENOMEM\ENOSPC in the guest? > >>>> The enforcement and handling, should be from the engine scope > >>>> and > >>>> not > >>>> from the Host perspective. > >>>> Actually the Host, should not be aware of Quota at all. > >>>>> Also what about cases of KSM\QCow2, disk\memory overcommit. > >>>> on QCOW issue, the active disk should consume the full potential > >>>> space > >>>> from the Quota, since we are not sure how much space will be in > >>>> use, > >>>> slthough the snapshot disk, will be updated to consume only its > >>>> real > >>>> size from the Quota. > >>>> > >>>> you can check the Enforcement section : > >>>> "When dealing with QCOW disks (which is not pre-allocated, like > >>>> templates or stateless VM) the Quota should consume the total > >>>> maximum > >>>> size of the disk, since it is the potential size that can be > >>>> used." > >>>> > >>>> for overcommit issue, please see CRUD section in the WIKI: > >>>> "...However, users will not be able to exceed the Quota > >>>> limitations > >>>> again after the resources are released." > >>>> > >>>>> Disk Templates. > >>>>> Storage for hibernation disk. > >>>>> Temporary and shared disks. > >>>> same logic as above (Enforcement section) > >>>>> Shared disks between VMs owned by different users. > >>>> Please see Dependencies / Related Features and Projects: > >>>> "When handling plug/unplug disks or attach/detach disks, the > >>>> entity will > >>>> still consume resources from its configured original Quota it > >>>> was > >>>> created on. " > >>>> > >>>> Which means the disk should consume from the same Quota all the > >>>> time > >>>> (not dependent on the users that use it). > >>>>> Backup snapshots (should they count in the quota? They are > >>>>> transient) > >>>> When ever a volume is created whether it is snapshot, backup > >>>> snapshot, > >>>> stateless disk, or any QCOW implementation, the enforcement > >>>> should > >>>> the > >>>> the same as described above (see Enforcement section) > >>>>> > >>>>> I also don't see how vcpu limiting is any good? I don't even > >>>>> know > >>>>> what > >>>>> it means. What happens in migration between hosts with > >>>>> different > >>>>> amount > >>>>> of physical CPUs? > >>>> The "atomic" section that Quota is handling in the run time > >>>> scope > >>>> is > >>>> cluster. > >>>> Actually for the user migration will be transparent since it is > >>>> consumed > >>>> from the same Quota, the only validation the VM should encounter > >>>> will be > >>>> the same as before in the Host perspective. > >>>> > >>>>> I also don't think CPU limiting is even a thing to be managed > >>>>> by > >>>>> a > >>>>> quota. There is no reason not to use 100% of the CPU if you are > >>>>> the only > >>>>> VM active. CPU scheduling should use a priority model and not a > >>>>> quota > >>>>> IMHO. > >>>> Again, the Quota should be managed from the engine level, and > >>>> should not > >>>> be reflected in the Host implementation. > >>>> Try to look at it, as an abstract management mechanism for > >>>> taking > >>>> notes > >>>> and managing resource consumes for the Administrator. > >>>> > >>>> A priority model is an interesting thought. > >>>> Now it can be supported, by using different grace percentage > >>>> from > >>>> one > >>>> Quota to another, or maybe create different Quota for Different > >>>> type of > >>>> users. > >>> > >>> So I understand there is a pure disconnect between host lever > >>> quotas and > >>> policies and this quota feature. > >>> Speaking with Livnat and Maor I see this is a good use case for > >>> an > >>> ovirt > >>> user to allow simple quotas to clients. > >>> > >>> It just feels like a feature that should be implemented as part > >>> of > >>> a > >>> grater policy structure validating user actions and not as a > >>> strict > >>> quota. > >>> > >>> > >>> I think snapshots and images should have different quotas where > >>> images > >>> are capped by size and snapshots are capped by number. > >>> > >>> Think of it from a hosting provider perspective. I want each > >>> client > >>> to > >>> get a maximum of 30GB of storage. > >>> I can limit them to 30GB. And let them calculated that if they > >>> made > >>> a > >>> 16GB worth of VM they can never snapshot it. On the other hand I > >>> can > >>> chose to limit them to 10GB and allow up to 2 snapshots to each > >>> clients. > >>> This way I know each client will take up to 30 GB and there will > >>> be > >>> no > >>> complaints when the user created a VM and now he can't snapshot > >>> it. > >>> Snapshotting can also be capped on the image level instead of the > >>> user > >>> level (3 snapshots per image). I tend to go towards the latter > >>> because > >>> it'll keep multiple disk VM snapshots from failing due to > >>> depleted > >>> quota. > >>> > >>> This will also make complaints about the pessimistic storage > >>> estimation > >>> invalid. > >> > >> _______________________________________________ > >> Engine-devel mailing list > >> Engine-devel at ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/engine-devel > >> > > From gjansen at redhat.com Sun Nov 27 18:01:49 2011 From: gjansen at redhat.com (Geert Jansen) Date: Sun, 27 Nov 2011 19:01:49 +0100 Subject: [Engine-devel] timeout when starting a new VM Message-ID: <4ED27B0D.7010403@redhat.com> Hi, i've got a setup with 1500 guests, of which about 800 are running now. Things start getting a little slow. I'm trying to start more guests via the ovirt-engine-sdk, but i get this error: File "/usr/bin/labmgr", line 171, in func_start vm.start(action) File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/brokers.py", line 2389, in start body=ParseHelper.toXml(action)) File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/proxy.py", line 42, in request conn=self.getConnectionsPool().getConnection()) File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/proxy.py", line 54, in __doRequest raise ConnectionError, str(e) ovirtsdk.infrastructure.errors.ConnectionError: [ERROR]::oVirt API connection failure, The read operation timed out Any ideas? Regards, Geert From gjansen at redhat.com Sun Nov 27 19:13:50 2011 From: gjansen at redhat.com (Geert Jansen) Date: Sun, 27 Nov 2011 20:13:50 +0100 Subject: [Engine-devel] SDK: multiple instances of API? Message-ID: <4ED28BEE.8050109@redhat.com> Hi Michael, when i try to create multiple instances of the API object like this: from ovirtsdk.api import API url = 'xxx' username = 'admin at internal' password = 'yyy' a1 = API(url, username, password) a2 = API(url, username, password) I get this error: Traceback (most recent call last): File "test.py", line 8, in a2 = API(url, username, password) File "/home/geertj/Source/ovirt-engine-sdk/src/ovirtsdk/api.py", line 47, in __init__ Mode.R) File "/home/geertj/Source/ovirt-engine-sdk/src/ovirtsdk/infrastructure/contextmanager.py", line 53, in add raise ImmutableError(key) ovirtsdk.infrastructure.errors.ImmutableError: [ERROR]::'proxy' is immutable. Any ideas? Thanks, Geert From gjansen at redhat.com Sun Nov 27 19:16:11 2011 From: gjansen at redhat.com (Geert Jansen) Date: Sun, 27 Nov 2011 20:16:11 +0100 Subject: [Engine-devel] SDK: multi threading Message-ID: <4ED28C7B.4060306@redhat.com> Hi Michael, two questions about threading..: - Is the SDK thread safe? - Can I make multiple concurrent requests to the API by using multiple threads, or do the threads block each other? Regards, Geert From mpastern at redhat.com Sun Nov 27 20:23:36 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Sun, 27 Nov 2011 22:23:36 +0200 Subject: [Engine-devel] SDK: multiple instances of API? In-Reply-To: <4ED28BEE.8050109@redhat.com> References: <4ED28BEE.8050109@redhat.com> Message-ID: <4ED29C48.6050205@redhat.com> On 11/27/2011 09:13 PM, Geert Jansen wrote: > Hi Michael, > > when i try to create multiple instances of the API object like this: > > from ovirtsdk.api import API > > url = 'xxx' > username = 'admin at internal' > password = 'yyy' > > a1 = API(url, username, password) > a2 = API(url, username, password) > > I get this error: > > Traceback (most recent call last): > File "test.py", line 8, in > a2 = API(url, username, password) > File "/home/geertj/Source/ovirt-engine-sdk/src/ovirtsdk/api.py", line 47, in __init__ > Mode.R) > File "/home/geertj/Source/ovirt-engine-sdk/src/ovirtsdk/infrastructure/contextmanager.py", line 53, in add > raise ImmutableError(key) > ovirtsdk.infrastructure.errors.ImmutableError: [ERROR]::'proxy' is immutable. > > Any ideas? this is another TODO, it remarked in code and in commit message. > > Thanks, > Geert > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel -- Michael Pasternak RedHat, ENG-Virtualization R&D From mpastern at redhat.com Sun Nov 27 20:25:52 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Sun, 27 Nov 2011 22:25:52 +0200 Subject: [Engine-devel] SDK: multi threading In-Reply-To: <4ED28C7B.4060306@redhat.com> References: <4ED28C7B.4060306@redhat.com> Message-ID: <4ED29CD0.3050002@redhat.com> On 11/27/2011 09:16 PM, Geert Jansen wrote: > Hi Michael, > > two questions about threading..: > > - Is the SDK thread safe? yes > > - Can I make multiple concurrent requests to the API by using multiple threads, or do the threads block each other? you can, each request will have own context. > > Regards, > Geert > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel -- Michael Pasternak RedHat, ENG-Virtualization R&D From mpastern at redhat.com Sun Nov 27 21:02:25 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Sun, 27 Nov 2011 23:02:25 +0200 Subject: [Engine-devel] timeout when starting a new VM In-Reply-To: <4ED27B0D.7010403@redhat.com> References: <4ED27B0D.7010403@redhat.com> Message-ID: <4ED2A561.3090903@redhat.com> On 11/27/2011 08:01 PM, Geert Jansen wrote: > Hi, > > i've got a setup with 1500 guests, of which about 800 are running now. Things start getting a little slow. I'm trying to start more guests via the ovirt-engine-sdk, but i > get this error: > > File "/usr/bin/labmgr", line 171, in func_start > vm.start(action) > File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/brokers.py", line 2389, in start > body=ParseHelper.toXml(action)) > File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/proxy.py", line 42, in request > conn=self.getConnectionsPool().getConnection()) > File "/usr/lib/python2.6/site-packages/ovirt_engine_sdk-1.0_SNAPSHOT-py2.6.egg/ovirtsdk/infrastructure/proxy.py", line 54, in __doRequest > raise ConnectionError, str(e) > ovirtsdk.infrastructure.errors.ConnectionError: [ERROR]::oVirt API connection failure, The read operation timed out this is known httplib issue, i wasn't able to reproduce it in 2.7, but i see you using 2.6 (as for now I'm yet not backported the code for 2.6) please file BZ for ovirt-engine-sdk, and as temporary workaround please try two things: 1) when you creating the proxy[1] you have [temeout] parameter in constructor, the default is 60 seconds, please raise it - it will solve any reconnection issues. [1] API(url, username, password, ..., timeout): 2) re-setting socket timeout to default import socket ... timeout = socket.getdefaulttimeout() socket.setdefaulttimeout(None) ...request()... socket.setdefaulttimeout(timeout) (let me know if it helped) > > Any ideas? > > Regards, > Geert > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel -- Michael Pasternak RedHat, ENG-Virtualization R&D From mpastern at redhat.com Mon Nov 28 14:10:16 2011 From: mpastern at redhat.com (Michael Pasternak) Date: Mon, 28 Nov 2011 16:10:16 +0200 Subject: [Engine-devel] Feedback on ovirt-engine-sdk In-Reply-To: <4ED1FFEA.20003@redhat.com> References: <4ED126AD.7000804@redhat.com> <4ED1FFEA.20003@redhat.com> Message-ID: <4ED39648.9060104@redhat.com> added support for actual resources as parameters for objects creation param = params.VM(name='my_vm', cluster=api.clusters.get(name='xxx'), template=api.templates.get(name='yyy'), ...) my_vm = api.vms.add(param) On 11/27/2011 11:16 AM, Michael Pasternak wrote: > > Hi Geert, > > On 11/26/2011 07:49 PM, Geert Jansen wrote: >> Hi Michael, >> >> i just wrote a small program with the new Python SDK. I found it very easy to use, so great work. I do have a few points of feedback: >> >> * Probably we want all Exceptions derived from a single base class. That way, the user can intercept any ovirtsdk with a single try/except clause. Currently that's not >> possible, as all your exceptions derive directly from Exception. > > sure, in my TODO list. > >> >> * I think the .get() method should be id-based not name based. Using "id" as a primary key has a number of advantages over name: >> >> - It is immutable, which makes e.g. renaming a lot easier. >> - It allows you to retrieve an object without a search query or a filter. >> - It matches how the API works as well. > > "id" is supported, you can do api.collection.get(id=xxx), only as you mentioned > "id" is not primary, but secondary key (i thought that searching by name will be easier), > but i can change it in a second if this is an issue. > >> >> Also currently i think it's not possible to retrieve a resource by its ID, unless you use the id= keyword argument to list() which retrieves the whole collection and then >> filters out the matching id. > > nope, it's supported - i even added it to examples.py [1] > -> vm5 = api.vms.get(id='02f0f4a4-9738-4731-83c4-293f3f734782') > > [1] will add it to get method .get(id, name) as you suggested. > >> >> * You may want to introduce some non-entity related operations, like ping(), and connect(). These allow you to better control when you connect, and allow an program to >> test if the connection details work. That helps client except the right exceptions at the right time, and display more relevant error messages. > > - no need for explicit connect() as its done implicitly on proxy construction > - no need for ping() as any attempt to use disconnected (and not recoverable) proxy, will cause ConnectionError, > so user should only /Trap/ this exception for disaster_recovery > >> >> * There's two Python objects involved for each API object. For a VM for example, you have params.VM and brokers.VM. This distinction is made visible to the user, for example: >> >> template = api.templates.get('Blank') >> >> vm = params.VM() >> vm.name = 'foo' >> vm.template = template # ERROR > > yep, you right doing this is in my TODO list as well > (had to switch to CLI, so i do have few leftovers[1] - this is one of them) > > [1] they all have simple workarounds (like this one as described below) > >> >> The last line need to be: >> >> vm.template = params.Template(id=template.id) > > for now (as workaround you can pass 'superclass', e.g vm.template = template.superclass), > but i suggest you just wait for another day till I'll support this. > >> >> In my version of the Python API i used mix-in classes to add the OO behavior to the generated classes. It was easy, because PyXB had support for that. This gave me the >> single class behavior. I am not sure if GenerateDS generated code allows that. I'm pretty sure it could be done though, but you may need to do some Python magic. > > no need, i do this with regular inheritance pattern > >> >> * Nitpick: should this just be called "ovirt" instead of "ovirtsdk"? > > initially it was, but since both cli & sdk would operate in same namespace, > i wanted to avoid future collisions and renamed them to ovirtsdk/ovirtcli. > >> >> Overall great work though! > > 10nx a lot mate. > >> >> Regards, >> Geert >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel at ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel > > -- Michael Pasternak RedHat, ENG-Virtualization R&D From mkublin at redhat.com Tue Nov 29 10:13:10 2011 From: mkublin at redhat.com (Michael Kublin) Date: Tue, 29 Nov 2011 05:13:10 -0500 (EST) Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: <9aa31dab-190e-4abe-84f4-acbdb22ea3ce@zmail14.collab.prod.int.phx2.redhat.com> Message-ID: <7fb0fa6e-74c4-4fa0-8aac-4f8b48d75ac0@zmail14.collab.prod.int.phx2.redhat.com> Hi All, We are introducing a new short term locking mechanism at engine. The motivation is races which are occurring between different flows, the main problem is : we did not update status of some entity and other flow was started , which is a main cause for different collisions and situation which can not be fixed in appropriate way by engine or vdsm. The current status is a workaround which is trying to reduce a race : in many places in the code, there is additional query to DB in order to check the appropriate status of entity. The proposed solution is internal locking mechanism - which will lock an appropriate entity until its status will not be updated in DB, after that an internal short term lock will be released. The design for locking mechanism can be found here: http://www.ovirt.org/wiki/Features/DetailedLockMechanism The patch which is introduce an implementation for mechanism can be found here: http://gerrit.ovirt.org/#change,403 (Using of new mechanism all around a code and integration with bll will be sent in the future, including cleaning of workaround) Regards Michael From ykaul at redhat.com Tue Nov 29 10:18:15 2011 From: ykaul at redhat.com (Yaniv Kaul) Date: Tue, 29 Nov 2011 12:18:15 +0200 Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: <7fb0fa6e-74c4-4fa0-8aac-4f8b48d75ac0@zmail14.collab.prod.int.phx2.redhat.com> References: <7fb0fa6e-74c4-4fa0-8aac-4f8b48d75ac0@zmail14.collab.prod.int.phx2.redhat.com> Message-ID: <4ED4B167.7000207@redhat.com> On 11/29/2011 12:13 PM, Michael Kublin wrote: > Hi All, > > We are introducing a new short term locking mechanism at engine. > The motivation is races which are occurring between different flows, the main problem is : > we did not update status of some entity and other flow was started , which is a main cause for different collisions and situation which > can not be fixed in appropriate way by engine or vdsm. > The current status is a workaround which is trying to reduce a race : in many places in the code, there is additional query to DB in order to check the appropriate status of entity. > The proposed solution is internal locking mechanism - which will lock an appropriate entity until its status will not be updated in DB, after that > an internal short term lock will be released. > > The design for locking mechanism can be found here: http://www.ovirt.org/wiki/Features/DetailedLockMechanism The page still has many remains of the template. > The patch which is introduce an implementation for mechanism can be found here: http://gerrit.ovirt.org/#change,403 > (Using of new mechanism all around a code and integration with bll will be sent in the future, including cleaning of workaround) > > Regards Michael > > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel There's an implicit assumption that there's only going to be a single active backend working against the database? Y. From mkublin at redhat.com Tue Nov 29 10:41:46 2011 From: mkublin at redhat.com (Michael Kublin) Date: Tue, 29 Nov 2011 05:41:46 -0500 (EST) Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: <4ED4B167.7000207@redhat.com> Message-ID: ----- Original Message ----- > From: "Yaniv Kaul" > To: "Michael Kublin" > Cc: engine-devel at ovirt.org > Sent: Tuesday, November 29, 2011 12:18:15 PM > Subject: Re: [Engine-devel] Requirement for Locking Mechanism > > On 11/29/2011 12:13 PM, Michael Kublin wrote: > > Hi All, > > > > We are introducing a new short term locking mechanism at engine. > > The motivation is races which are occurring between different > > flows, the main problem is : > > we did not update status of some entity and other flow was started > > , which is a main cause for different collisions and situation > > which > > can not be fixed in appropriate way by engine or vdsm. > > The current status is a workaround which is trying to reduce a race > > : in many places in the code, there is additional query to DB in > > order to check the appropriate status of entity. > > The proposed solution is internal locking mechanism - which will > > lock an appropriate entity until its status will not be updated in > > DB, after that > > an internal short term lock will be released. > > > > The design for locking mechanism can be found here: > > http://www.ovirt.org/wiki/Features/DetailedLockMechanism > > The page still has many remains of the template. > > > The patch which is introduce an implementation for mechanism can be > > found here: http://gerrit.ovirt.org/#change,403 > > (Using of new mechanism all around a code and integration with bll > > will be sent in the future, including cleaning of workaround) > > > > Regards Michael > > > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > There's an implicit assumption that there's only going to be a single > active backend working against the database? > Y. > Yes, the backend will lock some entity, until its status will not be updated in db, after that the lock will be released. So, it will solve a race between canDoAction and beginning of execute of the command From mgoldboi at redhat.com Tue Nov 29 11:40:03 2011 From: mgoldboi at redhat.com (Moran Goldboim) Date: Tue, 29 Nov 2011 13:40:03 +0200 Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: References: Message-ID: <4ED4C493.2030007@redhat.com> On 11/29/2011 12:41 PM, Michael Kublin wrote: > > ----- Original Message ----- >> From: "Yaniv Kaul" >> To: "Michael Kublin" >> Cc: engine-devel at ovirt.org >> Sent: Tuesday, November 29, 2011 12:18:15 PM >> Subject: Re: [Engine-devel] Requirement for Locking Mechanism >> >> On 11/29/2011 12:13 PM, Michael Kublin wrote: >>> Hi All, >>> >>> We are introducing a new short term locking mechanism at engine. >>> The motivation is races which are occurring between different >>> flows, the main problem is : >>> we did not update status of some entity and other flow was started >>> , which is a main cause for different collisions and situation >>> which >>> can not be fixed in appropriate way by engine or vdsm. >>> The current status is a workaround which is trying to reduce a race >>> : in many places in the code, there is additional query to DB in >>> order to check the appropriate status of entity. >>> The proposed solution is internal locking mechanism - which will >>> lock an appropriate entity until its status will not be updated in >>> DB, after that >>> an internal short term lock will be released. >>> >>> The design for locking mechanism can be found here: >>> http://www.ovirt.org/wiki/Features/DetailedLockMechanism >> The page still has many remains of the template. >> >>> The patch which is introduce an implementation for mechanism can be >>> found here: http://gerrit.ovirt.org/#change,403 >>> (Using of new mechanism all around a code and integration with bll >>> will be sent in the future, including cleaning of workaround) >>> >>> Regards Michael >>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >> There's an implicit assumption that there's only going to be a single >> active backend working against the database? >> Y. >> > Yes, the backend will lock some entity, until its status will not be updated in db, after that the lock will be released. > So, it will solve a race between canDoAction and beginning of execute of the command Michael, what do you think the performance implications on the system? > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From mkublin at redhat.com Tue Nov 29 12:50:56 2011 From: mkublin at redhat.com (Michael Kublin) Date: Tue, 29 Nov 2011 07:50:56 -0500 (EST) Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: <4ED4C493.2030007@redhat.com> Message-ID: <936a8ebd-51f4-47be-babe-0a11976c1f7d@zmail14.collab.prod.int.phx2.redhat.com> ----- Original Message ----- > From: "Moran Goldboim" > To: "Michael Kublin" > Cc: "Yaniv Kaul" , engine-devel at ovirt.org > Sent: Tuesday, November 29, 2011 1:40:03 PM > Subject: Re: [Engine-devel] Requirement for Locking Mechanism > > On 11/29/2011 12:41 PM, Michael Kublin wrote: > > > > ----- Original Message ----- > >> From: "Yaniv Kaul" > >> To: "Michael Kublin" > >> Cc: engine-devel at ovirt.org > >> Sent: Tuesday, November 29, 2011 12:18:15 PM > >> Subject: Re: [Engine-devel] Requirement for Locking Mechanism > >> > >> On 11/29/2011 12:13 PM, Michael Kublin wrote: > >>> Hi All, > >>> > >>> We are introducing a new short term locking mechanism at engine. > >>> The motivation is races which are occurring between different > >>> flows, the main problem is : > >>> we did not update status of some entity and other flow was > >>> started > >>> , which is a main cause for different collisions and situation > >>> which > >>> can not be fixed in appropriate way by engine or vdsm. > >>> The current status is a workaround which is trying to reduce a > >>> race > >>> : in many places in the code, there is additional query to DB in > >>> order to check the appropriate status of entity. > >>> The proposed solution is internal locking mechanism - which will > >>> lock an appropriate entity until its status will not be updated > >>> in > >>> DB, after that > >>> an internal short term lock will be released. > >>> > >>> The design for locking mechanism can be found here: > >>> http://www.ovirt.org/wiki/Features/DetailedLockMechanism > >> The page still has many remains of the template. > >> > >>> The patch which is introduce an implementation for mechanism can > >>> be > >>> found here: http://gerrit.ovirt.org/#change,403 > >>> (Using of new mechanism all around a code and integration with > >>> bll > >>> will be sent in the future, including cleaning of workaround) > >>> > >>> Regards Michael > >>> > >>> _______________________________________________ > >>> Engine-devel mailing list > >>> Engine-devel at ovirt.org > >>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >> There's an implicit assumption that there's only going to be a > >> single > >> active backend working against the database? > >> Y. > >> > > Yes, the backend will lock some entity, until its status will not > > be updated in db, after that the lock will be released. > > So, it will solve a race between canDoAction and beginning of > > execute of the command > > Michael, what do you think the performance implications on the > system? Because of locks are implemented in memory and because of today we already has some in memory lock mechanism (which is wrong) and after introducing of new mechanism we actually can reduce number of db queries, I think that performance will not decrease and maybe we will fill some improvements > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel at ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > From iheim at redhat.com Tue Nov 29 16:32:54 2011 From: iheim at redhat.com (Itamar Heim) Date: Tue, 29 Nov 2011 18:32:54 +0200 Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: References: Message-ID: <4ED50936.90307@redhat.com> On 11/29/2011 12:41 PM, Michael Kublin wrote: > > > ----- Original Message ----- >> From: "Yaniv Kaul" >> To: "Michael Kublin" >> Cc: engine-devel at ovirt.org >> Sent: Tuesday, November 29, 2011 12:18:15 PM >> Subject: Re: [Engine-devel] Requirement for Locking Mechanism >> >> On 11/29/2011 12:13 PM, Michael Kublin wrote: >>> Hi All, >>> >>> We are introducing a new short term locking mechanism at engine. >>> The motivation is races which are occurring between different >>> flows, the main problem is : >>> we did not update status of some entity and other flow was started >>> , which is a main cause for different collisions and situation >>> which >>> can not be fixed in appropriate way by engine or vdsm. >>> The current status is a workaround which is trying to reduce a race >>> : in many places in the code, there is additional query to DB in >>> order to check the appropriate status of entity. >>> The proposed solution is internal locking mechanism - which will >>> lock an appropriate entity until its status will not be updated in >>> DB, after that >>> an internal short term lock will be released. >>> >>> The design for locking mechanism can be found here: >>> http://www.ovirt.org/wiki/Features/DetailedLockMechanism >> >> The page still has many remains of the template. >> >>> The patch which is introduce an implementation for mechanism can be >>> found here: http://gerrit.ovirt.org/#change,403 >>> (Using of new mechanism all around a code and integration with bll >>> will be sent in the future, including cleaning of workaround) >>> >>> Regards Michael >>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel at ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >> >> There's an implicit assumption that there's only going to be a single >> active backend working against the database? >> Y. >> > Yes, the backend will lock some entity, until its status will not be updated in db, after that the lock will be released. > So, it will solve a race between canDoAction and beginning of execute of the command so how will distributed engines for scale out later on will look like with this mechanism? From lpeer at redhat.com Tue Nov 29 19:36:00 2011 From: lpeer at redhat.com (Livnat Peer) Date: Tue, 29 Nov 2011 21:36:00 +0200 Subject: [Engine-devel] Requirement for Locking Mechanism In-Reply-To: <4ED50936.90307@redhat.com> References: <4ED50936.90307@redhat.com> Message-ID: <4ED53420.8090601@redhat.com> On 11/29/2011 06:32 PM, Itamar Heim wrote: > On 11/29/2011 12:41 PM, Michael Kublin wrote: >> >> >> ----- Original Message ----- >>> From: "Yaniv Kaul" >>> To: "Michael Kublin" >>> Cc: engine-devel at ovirt.org >>> Sent: Tuesday, November 29, 2011 12:18:15 PM >>> Subject: Re: [Engine-devel] Requirement for Locking Mechanism >>> >>> On 11/29/2011 12:13 PM, Michael Kublin wrote: >>>> Hi All, >>>> >>>> We are introducing a new short term locking mechanism at engine. >>>> The motivation is races which are occurring between different >>>> flows, the main problem is : >>>> we did not update status of some entity and other flow was started >>>> , which is a main cause for different collisions and situation >>>> which >>>> can not be fixed in appropriate way by engine or vdsm. >>>> The current status is a workaround which is trying to reduce a race >>>> : in many places in the code, there is additional query to DB in >>>> order to check the appropriate status of entity. >>>> The proposed solution is internal locking mechanism - which will >>>> lock an appropriate entity until its status will not be updated in >>>> DB, after that >>>> an internal short term lock will be released. >>>> >>>> The design for locking mechanism can be found here: >>>> http://www.ovirt.org/wiki/Features/DetailedLockMechanism >>> >>> The page still has many remains of the template. >>> >>>> The patch which is introduce an implementation for mechanism can be >>>> found here: http://gerrit.ovirt.org/#change,403 >>>> (Using of new mechanism all around a code and integration with bll >>>> will be sent in the future, including cleaning of workaround) >>>> >>>> Regards Michael >>>> >>>> _______________________________________________ >>>> Engine-devel mailing list >>>> Engine-devel at ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>> >>> There's an implicit assumption that there's only going to be a single >>> active backend working against the database? >>> Y. >>> >> Yes, the backend will lock some entity, until its status will not be >> updated in db, after that the lock will be released. >> So, it will solve a race between canDoAction and beginning of execute >> of the command > > so how will distributed engines for scale out later on will look like > with this mechanism? We currently have caching in oVirt core in several places, when we'll handle distributed deployment we'll handle all of them. it can be either distributed cache or any other suitable solution. Livnat > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel From iheim at redhat.com Wed Nov 30 07:14:16 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 30 Nov 2011 09:14:16 +0200 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111129213604.GE13803@us.ibm.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> Message-ID: <4ED5D7C8.4090001@redhat.com> On 11/29/2011 11:36 PM, Adam Litke wrote: > On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >> * Adam Litke (agl at us.ibm.com) wrote: >>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>> >>>>> https://github.com/aglitke/vdsm-rest/ >>>>> >>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>> You can find the code on github. My goal is to eventually replace the current >>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>> entry point for direct virtualization management at the node level. >>>> >>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>> handled. Any VDSM developers willing to review this work? >>> >>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>> a version field to the root API object. As for compatibility, we'll just have >>> to decide on an API backwards-compat support policy. Would this be enough to >>> handle versioning issues? We shouldn't need anything like capabilities because >>> the API is discoverable. >> >> Right, that seems sensible. >> >> Did you find cases where RPC to REST resource mapping was difficult? > > I haven't yet fully implemented the current vdsm API but I suspect that certain > calls (like the ones you mention below) will require some extensions to what I > have available currently. The main missing piece is probably events and a nice > polling API. Another big piece of work will be to rebase onto the newly > redesigned storage model. > >> I could see something like migrate() plus migrateStatus() and >> migrateCancel() needing to add some kind of operational state that to the >> resource. And something like monitorCommand() which has both a possible >> side-effect and some freefrom response... > > Hopefully monitorCommand will not be too bad, since vdsm should be asking > libvirt for the VM details when they are needed. Of course we'll need to be > testing to make sure we aren't keeping state around. Also, I would expect > monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > libvirt). > >>>> >>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>> typically support async events and instead rely on polling of resources. I am >>>>> investigating what options are available for supporting async events via REST. >>>> >>>> I think typical is either polling or long polling. If it's a single >>>> resource, then perhaps long polling would be fine (won't be a large >>>> number of connections tied up if it's only a single resource). >>> >>> Not sure if this is what you are referring to, but I was thinking we could do a >>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>> event URIs. The server would respond with the status of these resources in one >>> response. I have some ideas on how we could wire up asynchronous events on the >>> server side to reduce the amount of actual work that such a batch-polling >>> operation would require. >> >> Oh, I just meant this: >> >> Polling (GET /event + 404 loop) >> Long polling (GET + block ... can chew up a thread connection) > > Yep. And we can talk later about building an API for efficient, repeated > polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST cc-ing engine-devel... > interface with event polling would be acceptable to them. It is critical that > we settle on an API that can become _the_ first-class vehicle for interacting > with vdsm. > > Thanks! > i have two points for consideration around this: 1. as the api between ovirt-engine and vdsm, I had a preference for the bus like nature of QMF, as it would allow multiple ovirt-engine to load balance handling of messages from the queue, and multiple consumers for some messages (say, history service picking up the stats in parallel to engine picking them, rather than copying them from engine). 2. as node level api, i think a lightweight ovirt-engine managing a single node and exposing the exact same REST API and behavior of the multi-node ovirt engine would be easier to cosnume for someone that wants to interact with a single node same way they would interact with ovirt-engine. From berrange at redhat.com Wed Nov 30 09:29:23 2011 From: berrange at redhat.com (Daniel P. Berrange) Date: Wed, 30 Nov 2011 09:29:23 +0000 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED5D7C8.4090001@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> Message-ID: <20111130092923.GC28621@redhat.com> On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > On 11/29/2011 11:36 PM, Adam Litke wrote: > > On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: > >> * Adam Litke (agl at us.ibm.com) wrote: > >>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: > >>>> * Adam Litke (agl at us.ibm.com) wrote: > >>>>> > >>>>> https://github.com/aglitke/vdsm-rest/ > >>>>> > >>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. > >>>>> You can find the code on github. My goal is to eventually replace the current > >>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to > >>>>> this new API. The major advantages to making this change are: 1) VDSM will gain > >>>>> a structured API that conceptually, structurally, and functionally aligns with > >>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an > >>>>> entry point for direct virtualization management at the node level. > >>>> > >>>> Adam, this looks like a nice PoC. I didn't see how API versioning is > >>>> handled. Any VDSM developers willing to review this work? > >>> > >>> Thanks for taking a look. I am not handling versioning yet. I think we can add > >>> a version field to the root API object. As for compatibility, we'll just have > >>> to decide on an API backwards-compat support policy. Would this be enough to > >>> handle versioning issues? We shouldn't need anything like capabilities because > >>> the API is discoverable. > >> > >> Right, that seems sensible. > >> > >> Did you find cases where RPC to REST resource mapping was difficult? > > > > I haven't yet fully implemented the current vdsm API but I suspect that certain > > calls (like the ones you mention below) will require some extensions to what I > > have available currently. The main missing piece is probably events and a nice > > polling API. Another big piece of work will be to rebase onto the newly > > redesigned storage model. > > > >> I could see something like migrate() plus migrateStatus() and > >> migrateCancel() needing to add some kind of operational state that to the > >> resource. And something like monitorCommand() which has both a possible > >> side-effect and some freefrom response... > > > > Hopefully monitorCommand will not be too bad, since vdsm should be asking > > libvirt for the VM details when they are needed. Of course we'll need to be > > testing to make sure we aren't keeping state around. Also, I would expect > > monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > > libvirt). > > > >>>> > >>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not > >>>>> typically support async events and instead rely on polling of resources. I am > >>>>> investigating what options are available for supporting async events via REST. > >>>> > >>>> I think typical is either polling or long polling. If it's a single > >>>> resource, then perhaps long polling would be fine (won't be a large > >>>> number of connections tied up if it's only a single resource). > >>> > >>> Not sure if this is what you are referring to, but I was thinking we could do a > >>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or > >>> event URIs. The server would respond with the status of these resources in one > >>> response. I have some ideas on how we could wire up asynchronous events on the > >>> server side to reduce the amount of actual work that such a batch-polling > >>> operation would require. > >> > >> Oh, I just meant this: > >> > >> Polling (GET /event + 404 loop) > >> Long polling (GET + block ... can chew up a thread connection) > > > > Yep. And we can talk later about building an API for efficient, repeated > > polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > cc-ing engine-devel... > > > interface with event polling would be acceptable to them. It is critical that > > we settle on an API that can become _the_ first-class vehicle for interacting > > with vdsm. > > i have two points for consideration around this: > 1. as the api between ovirt-engine and vdsm, I had a preference for the > bus like nature of QMF, as it would allow multiple ovirt-engine to load > balance handling of messages from the queue, and multiple consumers for > some messages (say, history service picking up the stats in parallel to > engine picking them, rather than copying them from engine). I tend to agree, using a bus like QMF between ovirt-engine and vdsm is an inherantly more scalable network architecture, since it avoids the need to have a direct point-to-point connection between the engine and every single node, instead you can build a resilient grid by stategically deploying QPid brokers. As compared to REST, it is also much better at coping with async events, since you can have a push, rather than pull, model which VDSM just puts events onto the bus as they're generated, to be lazily consumed by any remote nodes when desired. NB, I don't mean to imply that there should not *also* be a REST API for the node level. A REST API has the very compelling property that it is trivially accessible from anything with HTTP client support, which is basically everything in existance today. > 2. as node level api, i think a lightweight ovirt-engine managing a > single node and exposing the exact same REST API and behavior of the > multi-node ovirt engine would be easier to cosnume for someone that > wants to interact with a single node same way they would interact with > ovirt-engine. If the REST API is well specified, you wouldn't need to necessarily have a lightweight ovirt-engine deployed at the node level, you could just have VDSM implement the same REST specification natively, as is impl in the engine. Or perhaps provide a some shered code that can be plugged into both VDSM & the enegine to facilitate provision of the same REST interface. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| From iheim at redhat.com Wed Nov 30 13:21:07 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 30 Nov 2011 15:21:07 +0200 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130092923.GC28621@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130092923.GC28621@redhat.com> Message-ID: <4ED62DC3.2000606@redhat.com> On 11/30/2011 11:29 AM, Daniel P. Berrange wrote: > On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: >> On 11/29/2011 11:36 PM, Adam Litke wrote: >>> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>>>> >>>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>>> >>>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>>> entry point for direct virtualization management at the node level. >>>>>> >>>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>>> handled. Any VDSM developers willing to review this work? >>>>> >>>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>>> a version field to the root API object. As for compatibility, we'll just have >>>>> to decide on an API backwards-compat support policy. Would this be enough to >>>>> handle versioning issues? We shouldn't need anything like capabilities because >>>>> the API is discoverable. >>>> >>>> Right, that seems sensible. >>>> >>>> Did you find cases where RPC to REST resource mapping was difficult? >>> >>> I haven't yet fully implemented the current vdsm API but I suspect that certain >>> calls (like the ones you mention below) will require some extensions to what I >>> have available currently. The main missing piece is probably events and a nice >>> polling API. Another big piece of work will be to rebase onto the newly >>> redesigned storage model. >>> >>>> I could see something like migrate() plus migrateStatus() and >>>> migrateCancel() needing to add some kind of operational state that to the >>>> resource. And something like monitorCommand() which has both a possible >>>> side-effect and some freefrom response... >>> >>> Hopefully monitorCommand will not be too bad, since vdsm should be asking >>> libvirt for the VM details when they are needed. Of course we'll need to be >>> testing to make sure we aren't keeping state around. Also, I would expect >>> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >>> libvirt). >>> >>>>>> >>>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>>> typically support async events and instead rely on polling of resources. I am >>>>>>> investigating what options are available for supporting async events via REST. >>>>>> >>>>>> I think typical is either polling or long polling. If it's a single >>>>>> resource, then perhaps long polling would be fine (won't be a large >>>>>> number of connections tied up if it's only a single resource). >>>>> >>>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>>> event URIs. The server would respond with the status of these resources in one >>>>> response. I have some ideas on how we could wire up asynchronous events on the >>>>> server side to reduce the amount of actual work that such a batch-polling >>>>> operation would require. >>>> >>>> Oh, I just meant this: >>>> >>>> Polling (GET /event + 404 loop) >>>> Long polling (GET + block ... can chew up a thread connection) >>> >>> Yep. And we can talk later about building an API for efficient, repeated >>> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST >> >> cc-ing engine-devel... >> >>> interface with event polling would be acceptable to them. It is critical that >>> we settle on an API that can become _the_ first-class vehicle for interacting >>> with vdsm. >> >> i have two points for consideration around this: >> 1. as the api between ovirt-engine and vdsm, I had a preference for the >> bus like nature of QMF, as it would allow multiple ovirt-engine to load >> balance handling of messages from the queue, and multiple consumers for >> some messages (say, history service picking up the stats in parallel to >> engine picking them, rather than copying them from engine). > > I tend to agree, using a bus like QMF between ovirt-engine and vdsm is > an inherantly more scalable network architecture, since it avoids the > need to have a direct point-to-point connection between the engine and > every single node, instead you can build a resilient grid by stategically > deploying QPid brokers. > > As compared to REST, it is also much better at coping with async events, > since you can have a push, rather than pull, model which VDSM just puts > events onto the bus as they're generated, to be lazily consumed by any > remote nodes when desired. > > NB, I don't mean to imply that there should not *also* be a REST API for > the node level. A REST API has the very compelling property that it is > trivially accessible from anything with HTTP client support, which is > basically everything in existance today. > >> 2. as node level api, i think a lightweight ovirt-engine managing a >> single node and exposing the exact same REST API and behavior of the >> multi-node ovirt engine would be easier to cosnume for someone that >> wants to interact with a single node same way they would interact with >> ovirt-engine. > > If the REST API is well specified, you wouldn't need to necessarily have > a lightweight ovirt-engine deployed at the node level, you could just > have VDSM implement the same REST specification natively, as is impl > in the engine. Or perhaps provide a some shered code that can be plugged > into both VDSM& the enegine to facilitate provision of the same REST > interface. Indeed, but an API is not just the transport, it is also behavior and error code numbering, etc. trying to develop two different projects trying to do the same thing sounds like quite the overhead to me. I prefer investing the time in making a lighter version of the engine for a single node. then you would also get the web admin and power user portal if you want them. From agl at us.ibm.com Wed Nov 30 15:00:24 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 09:00:24 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED5D7C8.4090001@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> Message-ID: <20111130150024.GF13803@us.ibm.com> On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > On 11/29/2011 11:36 PM, Adam Litke wrote: > >On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: > >>* Adam Litke (agl at us.ibm.com) wrote: > >>>On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: > >>>>* Adam Litke (agl at us.ibm.com) wrote: > >>>>> > >>>>>https://github.com/aglitke/vdsm-rest/ > >>>>> > >>>>>Today I am releasing a proof of concept implementation of a REST API for vdsm. > >>>>>You can find the code on github. My goal is to eventually replace the current > >>>>>xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to > >>>>>this new API. The major advantages to making this change are: 1) VDSM will gain > >>>>>a structured API that conceptually, structurally, and functionally aligns with > >>>>>the ovirt-engine REST API, 2) this new API can be made public, thus providing an > >>>>>entry point for direct virtualization management at the node level. > >>>> > >>>>Adam, this looks like a nice PoC. I didn't see how API versioning is > >>>>handled. Any VDSM developers willing to review this work? > >>> > >>>Thanks for taking a look. I am not handling versioning yet. I think we can add > >>>a version field to the root API object. As for compatibility, we'll just have > >>>to decide on an API backwards-compat support policy. Would this be enough to > >>>handle versioning issues? We shouldn't need anything like capabilities because > >>>the API is discoverable. > >> > >>Right, that seems sensible. > >> > >>Did you find cases where RPC to REST resource mapping was difficult? > > > >I haven't yet fully implemented the current vdsm API but I suspect that certain > >calls (like the ones you mention below) will require some extensions to what I > >have available currently. The main missing piece is probably events and a nice > >polling API. Another big piece of work will be to rebase onto the newly > >redesigned storage model. > > > >>I could see something like migrate() plus migrateStatus() and > >>migrateCancel() needing to add some kind of operational state that to the > >>resource. And something like monitorCommand() which has both a possible > >>side-effect and some freefrom response... > > > >Hopefully monitorCommand will not be too bad, since vdsm should be asking > >libvirt for the VM details when they are needed. Of course we'll need to be > >testing to make sure we aren't keeping state around. Also, I would expect > >monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > >libvirt). > > > >>>> > >>>>>ovirt-engine wants to subscribe to asynchronous events. REST APIs do not > >>>>>typically support async events and instead rely on polling of resources. I am > >>>>>investigating what options are available for supporting async events via REST. > >>>> > >>>>I think typical is either polling or long polling. If it's a single > >>>>resource, then perhaps long polling would be fine (won't be a large > >>>>number of connections tied up if it's only a single resource). > >>> > >>>Not sure if this is what you are referring to, but I was thinking we could do a > >>>batch-polling mechanism where an API user passes in a list of task UUIDs and/or > >>>event URIs. The server would respond with the status of these resources in one > >>>response. I have some ideas on how we could wire up asynchronous events on the > >>>server side to reduce the amount of actual work that such a batch-polling > >>>operation would require. > >> > >>Oh, I just meant this: > >> > >>Polling (GET /event + 404 loop) > >>Long polling (GET + block ... can chew up a thread connection) > > > >Yep. And we can talk later about building an API for efficient, repeated > >polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > cc-ing engine-devel... > > >interface with event polling would be acceptable to them. It is critical that > >we settle on an API that can become _the_ first-class vehicle for interacting > >with vdsm. > > > >Thanks! > > > > > i have two points for consideration around this: > 1. as the api between ovirt-engine and vdsm, I had a preference for > the bus like nature of QMF, as it would allow multiple ovirt-engine > to load balance handling of messages from the queue, and multiple > consumers for some messages (say, history service picking up the > stats in parallel to engine picking them, rather than copying them > from engine). How easy is QMF to consume from a software development perspective? Would it be easy for someone to write a virsh-like tool against a QMF-based vdsm API? Would such a tool be able to run on multiple Linux distributions? > single node and exposing the exact same REST API and behavior of the > multi-node ovirt engine would be easier to cosnume for someone that > wants to interact with a single node same way they would interact > with ovirt-engine. In principle an ovirt-engine-light sounds great. Especially if we could get the existing GUIs for free. The main concern I have with this is the gargantuan nature of the ovirt-engine and GUI code bases. You want to reserve as many system resources for the guests but it seems that ovirt-engine will consume at least several GB of RAM and one or more CPU cores. How would you lighten up this JBoss stack without rewriting it in a more efficient language? One use case I am interested in is a single-node ovirt instance to control VMs on my laptop. I can spare several hundred MB of RAM at most for the management stack. -- Adam Litke IBM Linux Technology Center From emesika at redhat.com Wed Nov 30 15:06:37 2011 From: emesika at redhat.com (Eli Mesika) Date: Wed, 30 Nov 2011 10:06:37 -0500 (EST) Subject: [Engine-devel] Stable PCI Addresses design wiki Message-ID: http://www.ovirt.org/wiki/Features/Design/StablePCIAddresses From iheim at redhat.com Wed Nov 30 15:09:27 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 30 Nov 2011 17:09:27 +0200 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130150024.GF13803@us.ibm.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> Message-ID: <4ED64727.1010107@redhat.com> On 11/30/2011 05:00 PM, Adam Litke wrote: > On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: >> On 11/29/2011 11:36 PM, Adam Litke wrote: >>> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>>>> >>>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>>> >>>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>>> entry point for direct virtualization management at the node level. >>>>>> >>>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>>> handled. Any VDSM developers willing to review this work? >>>>> >>>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>>> a version field to the root API object. As for compatibility, we'll just have >>>>> to decide on an API backwards-compat support policy. Would this be enough to >>>>> handle versioning issues? We shouldn't need anything like capabilities because >>>>> the API is discoverable. >>>> >>>> Right, that seems sensible. >>>> >>>> Did you find cases where RPC to REST resource mapping was difficult? >>> >>> I haven't yet fully implemented the current vdsm API but I suspect that certain >>> calls (like the ones you mention below) will require some extensions to what I >>> have available currently. The main missing piece is probably events and a nice >>> polling API. Another big piece of work will be to rebase onto the newly >>> redesigned storage model. >>> >>>> I could see something like migrate() plus migrateStatus() and >>>> migrateCancel() needing to add some kind of operational state that to the >>>> resource. And something like monitorCommand() which has both a possible >>>> side-effect and some freefrom response... >>> >>> Hopefully monitorCommand will not be too bad, since vdsm should be asking >>> libvirt for the VM details when they are needed. Of course we'll need to be >>> testing to make sure we aren't keeping state around. Also, I would expect >>> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >>> libvirt). >>> >>>>>> >>>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>>> typically support async events and instead rely on polling of resources. I am >>>>>>> investigating what options are available for supporting async events via REST. >>>>>> >>>>>> I think typical is either polling or long polling. If it's a single >>>>>> resource, then perhaps long polling would be fine (won't be a large >>>>>> number of connections tied up if it's only a single resource). >>>>> >>>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>>> event URIs. The server would respond with the status of these resources in one >>>>> response. I have some ideas on how we could wire up asynchronous events on the >>>>> server side to reduce the amount of actual work that such a batch-polling >>>>> operation would require. >>>> >>>> Oh, I just meant this: >>>> >>>> Polling (GET /event + 404 loop) >>>> Long polling (GET + block ... can chew up a thread connection) >>> >>> Yep. And we can talk later about building an API for efficient, repeated >>> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST >> >> cc-ing engine-devel... >> >>> interface with event polling would be acceptable to them. It is critical that >>> we settle on an API that can become _the_ first-class vehicle for interacting >>> with vdsm. >>> >>> Thanks! >>> >> >> >> i have two points for consideration around this: >> 1. as the api between ovirt-engine and vdsm, I had a preference for >> the bus like nature of QMF, as it would allow multiple ovirt-engine >> to load balance handling of messages from the queue, and multiple >> consumers for some messages (say, history service picking up the >> stats in parallel to engine picking them, rather than copying them >> from engine). > > How easy is QMF to consume from a software development perspective? Would it be > easy for someone to write a virsh-like tool against a QMF-based vdsm API? Would > such a tool be able to run on multiple Linux distributions? it is supposed to have a cli console. cc'ing carl and Ted for more details. > >> single node and exposing the exact same REST API and behavior of the >> multi-node ovirt engine would be easier to cosnume for someone that >> wants to interact with a single node same way they would interact >> with ovirt-engine. > > In principle an ovirt-engine-light sounds great. Especially if we could get the > existing GUIs for free. The main concern I have with this is the gargantuan > nature of the ovirt-engine and GUI code bases. You want to reserve as many > system resources for the guests but it seems that ovirt-engine will consume at > least several GB of RAM and one or more CPU cores. How would you lighten up > this JBoss stack without rewriting it in a more efficient language? One use > case I am interested in is a single-node ovirt instance to control VMs on my > laptop. I can spare several hundred MB of RAM at most for the management stack. > this is much much lighter and faster with jboss7 (which iirc, takes 100MB to start, tomcat takes 80MB for comparison). then we can look at modularizing/alternatives if still needed. From agl at us.ibm.com Wed Nov 30 15:12:19 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 09:12:19 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130092923.GC28621@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130092923.GC28621@redhat.com> Message-ID: <20111130151219.GG13803@us.ibm.com> On Wed, Nov 30, 2011 at 09:29:23AM +0000, Daniel P. Berrange wrote: > On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > > On 11/29/2011 11:36 PM, Adam Litke wrote: > > > On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: > > >> * Adam Litke (agl at us.ibm.com) wrote: > > >>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: > > >>>> * Adam Litke (agl at us.ibm.com) wrote: > > >>>>> > > >>>>> https://github.com/aglitke/vdsm-rest/ > > >>>>> > > >>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. > > >>>>> You can find the code on github. My goal is to eventually replace the current > > >>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to > > >>>>> this new API. The major advantages to making this change are: 1) VDSM will gain > > >>>>> a structured API that conceptually, structurally, and functionally aligns with > > >>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an > > >>>>> entry point for direct virtualization management at the node level. > > >>>> > > >>>> Adam, this looks like a nice PoC. I didn't see how API versioning is > > >>>> handled. Any VDSM developers willing to review this work? > > >>> > > >>> Thanks for taking a look. I am not handling versioning yet. I think we can add > > >>> a version field to the root API object. As for compatibility, we'll just have > > >>> to decide on an API backwards-compat support policy. Would this be enough to > > >>> handle versioning issues? We shouldn't need anything like capabilities because > > >>> the API is discoverable. > > >> > > >> Right, that seems sensible. > > >> > > >> Did you find cases where RPC to REST resource mapping was difficult? > > > > > > I haven't yet fully implemented the current vdsm API but I suspect that certain > > > calls (like the ones you mention below) will require some extensions to what I > > > have available currently. The main missing piece is probably events and a nice > > > polling API. Another big piece of work will be to rebase onto the newly > > > redesigned storage model. > > > > > >> I could see something like migrate() plus migrateStatus() and > > >> migrateCancel() needing to add some kind of operational state that to the > > >> resource. And something like monitorCommand() which has both a possible > > >> side-effect and some freefrom response... > > > > > > Hopefully monitorCommand will not be too bad, since vdsm should be asking > > > libvirt for the VM details when they are needed. Of course we'll need to be > > > testing to make sure we aren't keeping state around. Also, I would expect > > > monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > > > libvirt). > > > > > >>>> > > >>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not > > >>>>> typically support async events and instead rely on polling of resources. I am > > >>>>> investigating what options are available for supporting async events via REST. > > >>>> > > >>>> I think typical is either polling or long polling. If it's a single > > >>>> resource, then perhaps long polling would be fine (won't be a large > > >>>> number of connections tied up if it's only a single resource). > > >>> > > >>> Not sure if this is what you are referring to, but I was thinking we could do a > > >>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or > > >>> event URIs. The server would respond with the status of these resources in one > > >>> response. I have some ideas on how we could wire up asynchronous events on the > > >>> server side to reduce the amount of actual work that such a batch-polling > > >>> operation would require. > > >> > > >> Oh, I just meant this: > > >> > > >> Polling (GET /event + 404 loop) > > >> Long polling (GET + block ... can chew up a thread connection) > > > > > > Yep. And we can talk later about building an API for efficient, repeated > > > polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > > > > cc-ing engine-devel... > > > > > interface with event polling would be acceptable to them. It is critical that > > > we settle on an API that can become _the_ first-class vehicle for interacting > > > with vdsm. > > > > i have two points for consideration around this: > > 1. as the api between ovirt-engine and vdsm, I had a preference for the > > bus like nature of QMF, as it would allow multiple ovirt-engine to load > > balance handling of messages from the queue, and multiple consumers for > > some messages (say, history service picking up the stats in parallel to > > engine picking them, rather than copying them from engine). > > I tend to agree, using a bus like QMF between ovirt-engine and vdsm is > an inherantly more scalable network architecture, since it avoids the > need to have a direct point-to-point connection between the engine and > every single node, instead you can build a resilient grid by stategically > deploying QPid brokers. > > As compared to REST, it is also much better at coping with async events, > since you can have a push, rather than pull, model which VDSM just puts > events onto the bus as they're generated, to be lazily consumed by any > remote nodes when desired. > > NB, I don't mean to imply that there should not *also* be a REST API for > the node level. A REST API has the very compelling property that it is > trivially accessible from anything with HTTP client support, which is > basically everything in existance today. My goal is to create a single first-class VDSM API that both ovirt-engine and external applications can use. I am ok with checking out QMF as long as it is distro agnostic and relatively light-weight. In that case, someone (maybe me) could always write a REST interface, but it would merely consume the first-class QMF API. I want to avoid having multiple, parallel APIs to maintain. > > 2. as node level api, i think a lightweight ovirt-engine managing a > > single node and exposing the exact same REST API and behavior of the > > multi-node ovirt engine would be easier to cosnume for someone that > > wants to interact with a single node same way they would interact with > > ovirt-engine. > > If the REST API is well specified, you wouldn't need to necessarily have > a lightweight ovirt-engine deployed at the node level, you could just > have VDSM implement the same REST specification natively, as is impl > in the engine. Or perhaps provide a some shered code that can be plugged > into both VDSM & the enegine to facilitate provision of the same REST > interface. Code sharing is hard between the engine and vdsm because the former is Java/C# and vdsm is python. It might be fine to simply expose a single QMF (or whatever) API from vdsm and let some better vdsm interface tools evolve from that (command-line shell, web UI, native app). -- Adam Litke IBM Linux Technology Center From agl at us.ibm.com Wed Nov 30 15:15:49 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 09:15:49 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED62DC3.2000606@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130092923.GC28621@redhat.com> <4ED62DC3.2000606@redhat.com> Message-ID: <20111130151549.GH13803@us.ibm.com> On Wed, Nov 30, 2011 at 03:21:07PM +0200, Itamar Heim wrote: > On 11/30/2011 11:29 AM, Daniel P. Berrange wrote: > >On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: > >>On 11/29/2011 11:36 PM, Adam Litke wrote: > >>>On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: > >>>>* Adam Litke (agl at us.ibm.com) wrote: > >>>>>On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: > >>>>>>* Adam Litke (agl at us.ibm.com) wrote: > >>>>>>> > >>>>>>>https://github.com/aglitke/vdsm-rest/ > >>>>>>> > >>>>>>>Today I am releasing a proof of concept implementation of a REST API for vdsm. > >>>>>>>You can find the code on github. My goal is to eventually replace the current > >>>>>>>xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to > >>>>>>>this new API. The major advantages to making this change are: 1) VDSM will gain > >>>>>>>a structured API that conceptually, structurally, and functionally aligns with > >>>>>>>the ovirt-engine REST API, 2) this new API can be made public, thus providing an > >>>>>>>entry point for direct virtualization management at the node level. > >>>>>> > >>>>>>Adam, this looks like a nice PoC. I didn't see how API versioning is > >>>>>>handled. Any VDSM developers willing to review this work? > >>>>> > >>>>>Thanks for taking a look. I am not handling versioning yet. I think we can add > >>>>>a version field to the root API object. As for compatibility, we'll just have > >>>>>to decide on an API backwards-compat support policy. Would this be enough to > >>>>>handle versioning issues? We shouldn't need anything like capabilities because > >>>>>the API is discoverable. > >>>> > >>>>Right, that seems sensible. > >>>> > >>>>Did you find cases where RPC to REST resource mapping was difficult? > >>> > >>>I haven't yet fully implemented the current vdsm API but I suspect that certain > >>>calls (like the ones you mention below) will require some extensions to what I > >>>have available currently. The main missing piece is probably events and a nice > >>>polling API. Another big piece of work will be to rebase onto the newly > >>>redesigned storage model. > >>> > >>>>I could see something like migrate() plus migrateStatus() and > >>>>migrateCancel() needing to add some kind of operational state that to the > >>>>resource. And something like monitorCommand() which has both a possible > >>>>side-effect and some freefrom response... > >>> > >>>Hopefully monitorCommand will not be too bad, since vdsm should be asking > >>>libvirt for the VM details when they are needed. Of course we'll need to be > >>>testing to make sure we aren't keeping state around. Also, I would expect > >>>monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for > >>>libvirt). > >>> > >>>>>> > >>>>>>>ovirt-engine wants to subscribe to asynchronous events. REST APIs do not > >>>>>>>typically support async events and instead rely on polling of resources. I am > >>>>>>>investigating what options are available for supporting async events via REST. > >>>>>> > >>>>>>I think typical is either polling or long polling. If it's a single > >>>>>>resource, then perhaps long polling would be fine (won't be a large > >>>>>>number of connections tied up if it's only a single resource). > >>>>> > >>>>>Not sure if this is what you are referring to, but I was thinking we could do a > >>>>>batch-polling mechanism where an API user passes in a list of task UUIDs and/or > >>>>>event URIs. The server would respond with the status of these resources in one > >>>>>response. I have some ideas on how we could wire up asynchronous events on the > >>>>>server side to reduce the amount of actual work that such a batch-polling > >>>>>operation would require. > >>>> > >>>>Oh, I just meant this: > >>>> > >>>>Polling (GET /event + 404 loop) > >>>>Long polling (GET + block ... can chew up a thread connection) > >>> > >>>Yep. And we can talk later about building an API for efficient, repeated > >>>polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST > >> > >>cc-ing engine-devel... > >> > >>>interface with event polling would be acceptable to them. It is critical that > >>>we settle on an API that can become _the_ first-class vehicle for interacting > >>>with vdsm. > >> > >>i have two points for consideration around this: > >>1. as the api between ovirt-engine and vdsm, I had a preference for the > >>bus like nature of QMF, as it would allow multiple ovirt-engine to load > >>balance handling of messages from the queue, and multiple consumers for > >>some messages (say, history service picking up the stats in parallel to > >>engine picking them, rather than copying them from engine). > > > >I tend to agree, using a bus like QMF between ovirt-engine and vdsm is > >an inherantly more scalable network architecture, since it avoids the > >need to have a direct point-to-point connection between the engine and > >every single node, instead you can build a resilient grid by stategically > >deploying QPid brokers. > > > >As compared to REST, it is also much better at coping with async events, > >since you can have a push, rather than pull, model which VDSM just puts > >events onto the bus as they're generated, to be lazily consumed by any > >remote nodes when desired. > > > >NB, I don't mean to imply that there should not *also* be a REST API for > >the node level. A REST API has the very compelling property that it is > >trivially accessible from anything with HTTP client support, which is > >basically everything in existance today. > > > >>2. as node level api, i think a lightweight ovirt-engine managing a > >>single node and exposing the exact same REST API and behavior of the > >>multi-node ovirt engine would be easier to cosnume for someone that > >>wants to interact with a single node same way they would interact with > >>ovirt-engine. > > > >If the REST API is well specified, you wouldn't need to necessarily have > >a lightweight ovirt-engine deployed at the node level, you could just > >have VDSM implement the same REST specification natively, as is impl > >in the engine. Or perhaps provide a some shered code that can be plugged > >into both VDSM& the enegine to facilitate provision of the same REST > >interface. > > Indeed, but an API is not just the transport, it is also behavior > and error code numbering, etc. > trying to develop two different projects trying to do the same thing > sounds like quite the overhead to me. Agreed. > I prefer investing the time in making a lighter version of the > engine for a single node. > then you would also get the web admin and power user portal if you > want them. Could you flesh out some details on the effort required to implement a light-weight engine? I am concerned that this is more complex than it seems. Also, how would we avoid dual maintenance between the full engine and the light one when a new feature/api is added? -- Adam Litke IBM Linux Technology Center From emesika at redhat.com Wed Nov 30 15:17:42 2011 From: emesika at redhat.com (Eli Mesika) Date: Wed, 30 Nov 2011 10:17:42 -0500 (EST) Subject: [Engine-devel] Stable PCI Addresses design wiki In-Reply-To: Message-ID: <8a9ce5ed-e449-414a-bb11-b3c189a15f2b@zmail13.collab.prod.int.phx2.redhat.com> Hi again The following is a design draft for a new feature of oVirt-engine planned for 3.1 The feature allow devices in guest virtual machines to retain the same PCI address allocations as other devices are added or removed from the guest configuration. This is particularly important for Windows guests in order to prevent warnings or reactivation when device addresses change. This feature is supported by libvirt and should be implemented by RHEVM and VDSM. When creating a VM, QEMU allocates PCI addresses to the guest devices, these addresses are being reported by libvirt to VDSM and VDSM should report it back to RHEVM. RHEVM should persist the PCI addresses and report it as part of the VM configuration on the next run. If a change to the VM devices occurred RHEVM should detect the change and persist the new PCI addresses. Please review. Thanks Eli Mesika Redhat ISRAEL ----- Original Message ----- > From: "Eli Mesika" > To: engine-devel at ovirt.org > Sent: Wednesday, November 30, 2011 5:06:37 PM > Subject: [Engine-devel] Stable PCI Addresses design wiki > > http://www.ovirt.org/wiki/Features/Design/StablePCIAddresses > _______________________________________________ > Engine-devel mailing list > Engine-devel at ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel > From iheim at redhat.com Wed Nov 30 15:18:47 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 30 Nov 2011 17:18:47 +0200 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130151549.GH13803@us.ibm.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130092923.GC28621@redhat.com> <4ED62DC3.2000606@redhat.com> <20111130151549.GH13803@us.ibm.com> Message-ID: <4ED64957.7040502@redhat.com> On 11/30/2011 05:15 PM, Adam Litke wrote: > On Wed, Nov 30, 2011 at 03:21:07PM +0200, Itamar Heim wrote: >> On 11/30/2011 11:29 AM, Daniel P. Berrange wrote: >>> On Wed, Nov 30, 2011 at 09:14:16AM +0200, Itamar Heim wrote: >>>> On 11/29/2011 11:36 PM, Adam Litke wrote: >>>>> On Tue, Nov 29, 2011 at 12:54:44PM -0800, Chris Wright wrote: >>>>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>>>> On Tue, Nov 29, 2011 at 11:39:08AM -0800, Chris Wright wrote: >>>>>>>> * Adam Litke (agl at us.ibm.com) wrote: >>>>>>>>> >>>>>>>>> https://github.com/aglitke/vdsm-rest/ >>>>>>>>> >>>>>>>>> Today I am releasing a proof of concept implementation of a REST API for vdsm. >>>>>>>>> You can find the code on github. My goal is to eventually replace the current >>>>>>>>> xmlrpc interface with a REST API. Once completed, ovirt-engine could switch to >>>>>>>>> this new API. The major advantages to making this change are: 1) VDSM will gain >>>>>>>>> a structured API that conceptually, structurally, and functionally aligns with >>>>>>>>> the ovirt-engine REST API, 2) this new API can be made public, thus providing an >>>>>>>>> entry point for direct virtualization management at the node level. >>>>>>>> >>>>>>>> Adam, this looks like a nice PoC. I didn't see how API versioning is >>>>>>>> handled. Any VDSM developers willing to review this work? >>>>>>> >>>>>>> Thanks for taking a look. I am not handling versioning yet. I think we can add >>>>>>> a version field to the root API object. As for compatibility, we'll just have >>>>>>> to decide on an API backwards-compat support policy. Would this be enough to >>>>>>> handle versioning issues? We shouldn't need anything like capabilities because >>>>>>> the API is discoverable. >>>>>> >>>>>> Right, that seems sensible. >>>>>> >>>>>> Did you find cases where RPC to REST resource mapping was difficult? >>>>> >>>>> I haven't yet fully implemented the current vdsm API but I suspect that certain >>>>> calls (like the ones you mention below) will require some extensions to what I >>>>> have available currently. The main missing piece is probably events and a nice >>>>> polling API. Another big piece of work will be to rebase onto the newly >>>>> redesigned storage model. >>>>> >>>>>> I could see something like migrate() plus migrateStatus() and >>>>>> migrateCancel() needing to add some kind of operational state that to the >>>>>> resource. And something like monitorCommand() which has both a possible >>>>>> side-effect and some freefrom response... >>>>> >>>>> Hopefully monitorCommand will not be too bad, since vdsm should be asking >>>>> libvirt for the VM details when they are needed. Of course we'll need to be >>>>> testing to make sure we aren't keeping state around. Also, I would expect >>>>> monitorCommand to 'taint' the VM in the eyes of the vdsm API (as it does for >>>>> libvirt). >>>>> >>>>>>>> >>>>>>>>> ovirt-engine wants to subscribe to asynchronous events. REST APIs do not >>>>>>>>> typically support async events and instead rely on polling of resources. I am >>>>>>>>> investigating what options are available for supporting async events via REST. >>>>>>>> >>>>>>>> I think typical is either polling or long polling. If it's a single >>>>>>>> resource, then perhaps long polling would be fine (won't be a large >>>>>>>> number of connections tied up if it's only a single resource). >>>>>>> >>>>>>> Not sure if this is what you are referring to, but I was thinking we could do a >>>>>>> batch-polling mechanism where an API user passes in a list of task UUIDs and/or >>>>>>> event URIs. The server would respond with the status of these resources in one >>>>>>> response. I have some ideas on how we could wire up asynchronous events on the >>>>>>> server side to reduce the amount of actual work that such a batch-polling >>>>>>> operation would require. >>>>>> >>>>>> Oh, I just meant this: >>>>>> >>>>>> Polling (GET /event + 404 loop) >>>>>> Long polling (GET + block ... can chew up a thread connection) >>>>> >>>>> Yep. And we can talk later about building an API for efficient, repeated >>>>> polling. I wonder if the ovirt-engine guys could weigh in as to whether a REST >>>> >>>> cc-ing engine-devel... >>>> >>>>> interface with event polling would be acceptable to them. It is critical that >>>>> we settle on an API that can become _the_ first-class vehicle for interacting >>>>> with vdsm. >>>> >>>> i have two points for consideration around this: >>>> 1. as the api between ovirt-engine and vdsm, I had a preference for the >>>> bus like nature of QMF, as it would allow multiple ovirt-engine to load >>>> balance handling of messages from the queue, and multiple consumers for >>>> some messages (say, history service picking up the stats in parallel to >>>> engine picking them, rather than copying them from engine). >>> >>> I tend to agree, using a bus like QMF between ovirt-engine and vdsm is >>> an inherantly more scalable network architecture, since it avoids the >>> need to have a direct point-to-point connection between the engine and >>> every single node, instead you can build a resilient grid by stategically >>> deploying QPid brokers. >>> >>> As compared to REST, it is also much better at coping with async events, >>> since you can have a push, rather than pull, model which VDSM just puts >>> events onto the bus as they're generated, to be lazily consumed by any >>> remote nodes when desired. >>> >>> NB, I don't mean to imply that there should not *also* be a REST API for >>> the node level. A REST API has the very compelling property that it is >>> trivially accessible from anything with HTTP client support, which is >>> basically everything in existance today. >>> >>>> 2. as node level api, i think a lightweight ovirt-engine managing a >>>> single node and exposing the exact same REST API and behavior of the >>>> multi-node ovirt engine would be easier to cosnume for someone that >>>> wants to interact with a single node same way they would interact with >>>> ovirt-engine. >>> >>> If the REST API is well specified, you wouldn't need to necessarily have >>> a lightweight ovirt-engine deployed at the node level, you could just >>> have VDSM implement the same REST specification natively, as is impl >>> in the engine. Or perhaps provide a some shered code that can be plugged >>> into both VDSM& the enegine to facilitate provision of the same REST >>> interface. >> >> Indeed, but an API is not just the transport, it is also behavior >> and error code numbering, etc. >> trying to develop two different projects trying to do the same thing >> sounds like quite the overhead to me. > > Agreed. > >> I prefer investing the time in making a lighter version of the >> engine for a single node. >> then you would also get the web admin and power user portal if you >> want them. > > Could you flesh out some details on the effort required to implement a > light-weight engine? I am concerned that this is more complex than it seems. > Also, how would we avoid dual maintenance between the full engine and the light > one when a new feature/api is added? > I would use the same engine, moduling out parts not relevant for single node. first we need to move to JBoss AS 7 though to get it to be lighter, then we can look at other angles (make parts of the engine optional, allow it to run with/without jboss, etc.) From gjansen at redhat.com Wed Nov 30 15:32:41 2011 From: gjansen at redhat.com (Geert Jansen) Date: Wed, 30 Nov 2011 16:32:41 +0100 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64727.1010107@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> Message-ID: <4ED64C99.50706@redhat.com> On 11/30/2011 04:09 PM, Itamar Heim wrote: >> How easy is QMF to consume from a software development perspective? >> Would it be >> easy for someone to write a virsh-like tool against a QMF-based vdsm >> API? Would >> such a tool be able to run on multiple Linux distributions? > > it is supposed to have a cli console. > cc'ing carl and Ted for more details. I'm not sure how hard it is technically. But for ISV's, I can tell you that almost nobody has experience with it. But, is this intended to be a public API though? Regards, Geert From iheim at redhat.com Wed Nov 30 15:34:55 2011 From: iheim at redhat.com (Itamar Heim) Date: Wed, 30 Nov 2011 17:34:55 +0200 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64C99.50706@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64C99.50706@redhat.com> Message-ID: <4ED64D1F.2020305@redhat.com> On 11/30/2011 05:32 PM, Geert Jansen wrote: > > On 11/30/2011 04:09 PM, Itamar Heim wrote: > >>> How easy is QMF to consume from a software development perspective? >>> Would it be >>> easy for someone to write a virsh-like tool against a QMF-based vdsm >>> API? Would >>> such a tool be able to run on multiple Linux distributions? >> >> it is supposed to have a cli console. >> cc'ing carl and Ted for more details. > > I'm not sure how hard it is technically. But for ISV's, I can tell you > that almost nobody has experience with it. > > But, is this intended to be a public API though? yes, for VDSM. but if we go with ovirt-lite for single node, they will get exactly the same REST API as the full engine has. otherwise they will get a "similar" API, which meand double coding/testing/etc. btw, i think QMF has a REST bridge, but still it will be a different api than going the engine-lite approach. From agl at us.ibm.com Wed Nov 30 15:50:06 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 09:50:06 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64C99.50706@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64C99.50706@redhat.com> Message-ID: <20111130155006.GI13803@us.ibm.com> On Wed, Nov 30, 2011 at 04:32:41PM +0100, Geert Jansen wrote: > > On 11/30/2011 04:09 PM, Itamar Heim wrote: > > >>How easy is QMF to consume from a software development perspective? > >>Would it be > >>easy for someone to write a virsh-like tool against a QMF-based vdsm > >>API? Would > >>such a tool be able to run on multiple Linux distributions? > > > >it is supposed to have a cli console. > >cc'ing carl and Ted for more details. > > I'm not sure how hard it is technically. But for ISV's, I can tell > you that almost nobody has experience with it. > > But, is this intended to be a public API though? Yes! Vdsm already has a 'private' API that only RHEV-m used to consume. In the ovirt world, vdsm needs to expose a real public API (by whatever means we choose) so that external entities (ISVs) can leverage vdsm directly if they so choose. This API must be the same one that ovirt-engine uses, otherwise we are maintaining two incompatible APIs. -- Adam Litke IBM Linux Technology Center From agl at us.ibm.com Wed Nov 30 15:52:37 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 09:52:37 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64C99.50706@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64C99.50706@redhat.com> Message-ID: <20111130155237.GJ13803@us.ibm.com> On Wed, Nov 30, 2011 at 04:32:41PM +0100, Geert Jansen wrote: > > On 11/30/2011 04:09 PM, Itamar Heim wrote: > > >>How easy is QMF to consume from a software development perspective? > >>Would it be > >>easy for someone to write a virsh-like tool against a QMF-based vdsm > >>API? Would > >>such a tool be able to run on multiple Linux distributions? > > > >it is supposed to have a cli console. > >cc'ing carl and Ted for more details. > > I'm not sure how hard it is technically. But for ISV's, I can tell > you that almost nobody has experience with it. That is not a good sign. Certainly there must be some sort of standardized and well understood API transport that we can use. We're not doing anything particularly novel as far as the API is concerned. -- Adam Litke IBM Linux Technology Center From berrange at redhat.com Wed Nov 30 15:54:34 2011 From: berrange at redhat.com (Daniel P. Berrange) Date: Wed, 30 Nov 2011 15:54:34 +0000 Subject: [Engine-devel] Stable PCI Addresses design wiki In-Reply-To: References: Message-ID: <20111130155434.GB1160@redhat.com> On Wed, Nov 30, 2011 at 10:06:37AM -0500, Eli Mesika wrote: > http://www.ovirt.org/wiki/Features/Design/StablePCIAddresses My primary comment on this is that you likely don't want to restrict yourself to PCI addresses. If RHEVM intends to use virtio-serial controllers you want to maintain stable virtio serial addresses If RHEVM intends to use CCID smartcard controllers you want to maintain stable CCID device addresses If RHEVM intends to use SCSI controllers you will want to maintain SCSI drive addresses If RHEVM intends to use USB controllers you will want to maintain USB device addresses I think you get the idea :-) In general you can say that every single device listed in the XML will ultimately have an address associated with it. The type address will vary depending on what type of controller the device is attached to. In addition, when you start dealing with these other non-PCI address types, you will also need to start dealing with controller devices. eg, if you add a SCSI disk to the XML First of all libvirt will want to assign an address to this drive
Then, if the corresponding controller does not exist already, libvirt will auto-add a controller device, which itself has an address you will need to track:
In addition, when QEMU gains support for PCI bridges, or multiple PCI root complexes, there will also be the possibility of dealing with multiple PCI controllers in the XML too. So even if you only implement PCI address support in the first iteration, I'd really encourage you to at least consider how you will cope with the other address types sooner, rather than later. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| From anthony at codemonkey.ws Wed Nov 30 15:44:19 2011 From: anthony at codemonkey.ws (Anthony Liguori) Date: Wed, 30 Nov 2011 09:44:19 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64727.1010107@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> Message-ID: <4ED64F53.5020801@codemonkey.ws> On 11/30/2011 09:09 AM, Itamar Heim wrote: > On 11/30/2011 05:00 PM, Adam Litke wrote: >>> single node and exposing the exact same REST API and behavior of the >>> multi-node ovirt engine would be easier to cosnume for someone that >>> wants to interact with a single node same way they would interact >>> with ovirt-engine. >> >> In principle an ovirt-engine-light sounds great. Especially if we could get the >> existing GUIs for free. The main concern I have with this is the gargantuan >> nature of the ovirt-engine and GUI code bases. You want to reserve as many >> system resources for the guests but it seems that ovirt-engine will consume at >> least several GB of RAM and one or more CPU cores. How would you lighten up >> this JBoss stack without rewriting it in a more efficient language? One use >> case I am interested in is a single-node ovirt instance to control VMs on my >> laptop. I can spare several hundred MB of RAM at most for the management stack. >> > > this is much much lighter and faster with jboss7 (which iirc, takes > 100MB to start, tomcat takes 80MB for comparison). When thinking about something like ovirt-node, my concern is that a jboss7 based ovirt-lite would end up significantly impact the size of the image. I think ovirt-node tends to be around ~100MB today which includes VDSM. A simple REST API (as Adam proposes) wouldn't increase this size at all. Just the openjdk package looks to be around 72MB. Regards, Anthony Liguori From mlipchuk at redhat.com Wed Nov 30 16:00:09 2011 From: mlipchuk at redhat.com (Maor) Date: Wed, 30 Nov 2011 18:00:09 +0200 Subject: [Engine-devel] Quota design page In-Reply-To: <8a9ce5ed-e449-414a-bb11-b3c189a15f2b@zmail13.collab.prod.int.phx2.redhat.com> References: <8a9ce5ed-e449-414a-bb11-b3c189a15f2b@zmail13.collab.prod.int.phx2.redhat.com> Message-ID: <4ED65309.4090001@redhat.com> Hi everyone, Following the introduction of Quota feature description, The following link is the design of Quota feature: http://www.ovirt.org/wiki/Features/Design/Quota Please feel free, to share your comments. Thank you, Maor From mlipchuk at redhat.com Wed Nov 30 16:01:34 2011 From: mlipchuk at redhat.com (Maor) Date: Wed, 30 Nov 2011 18:01:34 +0200 Subject: [Engine-devel] Quota feature design Message-ID: <4ED6535E.4020404@redhat.com> Hi everyone, Following the introduction of Quota feature description, The following link is the design of Quota feature: http://www.ovirt.org/wiki/Features/Design/Quota Please feel free, to share your comments. Thank you, Maor From berrange at redhat.com Wed Nov 30 16:22:48 2011 From: berrange at redhat.com (Daniel P. Berrange) Date: Wed, 30 Nov 2011 16:22:48 +0000 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64F53.5020801@codemonkey.ws> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64F53.5020801@codemonkey.ws> Message-ID: <20111130162248.GA1555@redhat.com> On Wed, Nov 30, 2011 at 09:44:19AM -0600, Anthony Liguori wrote: > On 11/30/2011 09:09 AM, Itamar Heim wrote: > > On 11/30/2011 05:00 PM, Adam Litke wrote: > >>> single node and exposing the exact same REST API and behavior of the > >>> multi-node ovirt engine would be easier to cosnume for someone that > >>> wants to interact with a single node same way they would interact > >>> with ovirt-engine. > >> > >> In principle an ovirt-engine-light sounds great. Especially if we could get the > >> existing GUIs for free. The main concern I have with this is the gargantuan > >> nature of the ovirt-engine and GUI code bases. You want to reserve as many > >> system resources for the guests but it seems that ovirt-engine will consume at > >> least several GB of RAM and one or more CPU cores. How would you lighten up > >> this JBoss stack without rewriting it in a more efficient language? One use > >> case I am interested in is a single-node ovirt instance to control VMs on my > >> laptop. I can spare several hundred MB of RAM at most for the management stack. > >> > > > > this is much much lighter and faster with jboss7 (which iirc, takes > > 100MB to start, tomcat takes 80MB for comparison). > > When thinking about something like ovirt-node, my concern is that a jboss7 based > ovirt-lite would end up significantly impact the size of the image. > > I think ovirt-node tends to be around ~100MB today which includes VDSM. A > simple REST API (as Adam proposes) wouldn't increase this size at all. > > Just the openjdk package looks to be around 72MB. IIUC, one of the goals of VDSM is to be a general purpose reusable node agent for non-oVirt management engines to consume too. IME, to maximise the chances of success for broad adoption, simplicity & flexibility are key. Every piece of extra infrastructure a project mandates will reduce the scope for where it can be used. The idea of deploying an ovirt-engine-light just to get a REST based interface to VDSM just doesn't fit in with the simple & flexible approach you need to succeed. Instead of just installing a simple python package, you now need to deploy an entire java app server, and application ontop. Furthermore users who are python developers will be turned off by the idea ofhaving to deal with java, while java developers will be turned off by the idea of relying on this python bit at the bottom of the stack. Similarly if VDSM mandated use of AMQP for all its communication, this would be a turn off for usage from applications which already have a message bus/transport which is not AMQP based. So, IMHO, if VDSM wants to be easily consumable by other non-oVirt based systems, having a native REST API directly implemented in VDSM code is a critical factor to its success. I don't think this is mutually exclusive with having AMQP used for comms between VDSM & ovirt-engine. One possible implementation strategy would be to have a 'vdsm-qmf' agent running on the node which talks REST to VDSM and translate it to QMF & vica-verca. Or you could have an optional plugin for VDSM which enabled AQMP as an alternative to REST. The key is really to ensure that VDSM retains the ability to be deployed with minimal mandatory infrastructure burden/reqiurements. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| From chrisw at sous-sol.org Wed Nov 30 16:45:32 2011 From: chrisw at sous-sol.org (Chris Wright) Date: Wed, 30 Nov 2011 08:45:32 -0800 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130162248.GA1555@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64F53.5020801@codemonkey.ws> <20111130162248.GA1555@redhat.com> Message-ID: <20111130164532.GE13696@sequoia.sous-sol.org> * Daniel P. Berrange (berrange at redhat.com) wrote: > I don't think this is mutually exclusive with having AMQP used for comms > between VDSM & ovirt-engine. One possible implementation strategy would > be to have a 'vdsm-qmf' agent running on the node which talks REST to > VDSM and translate it to QMF & vica-verca. Or you could have an optional > plugin for VDSM which enabled AQMP as an alternative to REST. Right, there are a number of AMQP <-> REST projects out there, so shouldn't be mutually exclusive. > The key is really to ensure that VDSM retains the ability to be deployed > with minimal mandatory infrastructure burden/reqiurements. I completely agree. thanks, -chris From agl at us.ibm.com Wed Nov 30 19:28:06 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 13:28:06 -0600 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130162248.GA1555@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64F53.5020801@codemonkey.ws> <20111130162248.GA1555@redhat.com> Message-ID: <20111130192806.GL13803@us.ibm.com> We had a very useful discussion on IRC regarding these ideas. Here is that transcript for those of you who are interested: aglitke: I have, thanks to Chris Write I know I should chime in, as well as smizrahi I suppose. yep... Would be good to get some additional perspectives. QMF is being pushed hard at the moment. Or an ovirt-engine light (which I am dubious of) Well, I'm really for QMF (or anything else that supports messaging. We can't scale if we have to be polled for everything I guess I don't understand the multiple ovirt-engine use case? When would multple engines control the same node at the same time? rest can support notifications there's nothing that says that an HTTP request has to return in any amount of time If you need a bus in the engine, just write a QMF broker in your code that consumes the rest APi you can have an event collection, and fetch it using that as a mechanism to receive async events Then you can fill your engine code with QMF consoles to spread those events around. it's certainly not pub/sub, but you don't have to do polling aliguori: http requests do time out. You actually do want them to time out. Also we want to announce things like "network disappeared" and "machine crashed which shouldn't just have a request waiting for them. with real web servers over proxies, yes aglitke: sure you could certainly write the QMF agent for the node which talks REST to VDSM you could even argue that would be good because it would keep VDSM de-coupled from the messaging service as long as the QMF agent & VDSM were on the same node most of the performance questions around REST vs QMF would then be irrelevant, because REST usage would never stray outside the local node for the ovirt-engine <-> vdsm comms path indeed well, as the main consumer of the API would never use the REST API. Why is it even good? Just have everyone use AMF QMF well it depends on what other consumers of VDSM are around I don;t think that any consumer looking to scale would rather use the REST API or whether VDSM were intended to be used standarlone outside RHEVM or were expecting to be easily integrated with other mgmt systems We would like to be able to use vdsm standalone as a basic requirement. danpb: I would love to see it used outside ovirt-engine There are cases where not all ovirt consumers will want all of the pieces. The point being, if ovirt-engine found it necessary to use QMF, why shouldn't any other users eg, if some other 3rd party came along and wanted to use VDSM in their app, but already has a comms system/bus that is not AMQP, then integration will liekly be easier via REST apart from the obvious simplicity of REST danpb: you do have a point there smizrahi: IME simplicity of use is key to getting adoption - every piece of extra infrastructure you mandate, reduces your pool of consumers a good % of people will simply not even consider VDSM if it mandated QMF, whereas everyone in today's world can trivially use REST Also, ovirt-engine can modularize their consumption via QMF such that other projects could pick that up if they wanted to use it. It's all about building blocks in my mind. I see how the AMF proxy xould just be on the host QMF and we'd have a dedicated collectEvents API call yes. I also have some ideas on how to make this interface efficient in vdsm you could create a resource called an event sink or event monitor this would essentially be a list of events and task ids you want to watch. Ho will async operation work. We would like to have all operations be asynchronous How when this is created, it tells vdsm to set up some internal async handlers to cache the status of these tasks/events. yea but them shotrt living tasks have a large overhead as they are bound by the minimum collect interval For those, you would just use the standard polling aliguori suggested having a single. global event / task source that you could connect to. danpb suggested using keepalive or chunking smizrahi, so a QMF -> REST bridge wouldn't be a bad thing either IMHO We could technically optimize single host scenarios with a polling a pipe hinting you to collet i don't really care how things are implemented but QMF is an obscure protocol today if we did this, the server could send out things as they happen. REST is becoming the defacto standard management interface I would like to get abarons point of view. Too bad he missed this. I can post it to the mailing list. Do we have a marginal consensus among those of us participating here? smizrahi, REST is really a compromise, if it wasn't REST, I'd be advocating CIM ;-) yuck I would still like to get event handling nailed down before we go full throttle on this but I would rather it happening on the mailing lists as all the bigshots are not here Yeah, I'll try to take a stab at summarizing the thoughts here. Then I will post to the list for comments with a more thorough writeup... aglitke: excellent Summary of potential architecture: 1. VDSM adopts a REST API as the lowest level API 2. we create a QMF broker that consumes the rest API and exposes a bus to the engine 3. The engine can add any number of QMF consoles to this bus for things like stats, logging, reports, etc. 4. ISVs may interact with vdsm directly via its REST API Outstanding issues: 1. We must flesh out how events will be presented in the REST API (including the mechanism) ... aliguori, danpb, smizrahi: Agreed on these high-level points? aglitke: yeah, that's a viable plan to me ack One of the things I worry is problems with putting ovirt-engine specific logic in the QMF brgide I would like to stress that the bridge has to be ovirt-engine specific or at least to the same messaging API -- Adam Litke IBM Linux Technology Center From cctrieloff at redhat.com Wed Nov 30 21:54:51 2011 From: cctrieloff at redhat.com (Carl Trieloff) Date: Wed, 30 Nov 2011 16:54:51 -0500 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED64727.1010107@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> Message-ID: <4ED6A62B.7050407@redhat.com> On 11/30/2011 10:09 AM, Itamar Heim wrote: >> How easy is QMF to consume from a software development perspective? >> Would it be >> easy for someone to write a virsh-like tool against a QMF-based vdsm >> API? Would >> such a tool be able to run on multiple Linux distributions? > > it is supposed to have a cli console. > cc'ing carl and Ted for more details. yes it is dirt easy, and there are also a bunch of generic tools to script against QMF with. i.e. you can call just about anything in QMF with about 4-5 lines of script. Carl. From cctrieloff at redhat.com Wed Nov 30 21:56:46 2011 From: cctrieloff at redhat.com (Carl Trieloff) Date: Wed, 30 Nov 2011 16:56:46 -0500 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130151219.GG13803@us.ibm.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130092923.GC28621@redhat.com> <20111130151219.GG13803@us.ibm.com> Message-ID: <4ED6A69E.4090005@redhat.com> On 11/30/2011 10:12 AM, Adam Litke wrote: > My goal is to create a single first-class VDSM API that both ovirt-engine and > external applications can use. I am ok with checking out QMF as long as it is > distro agnostic and relatively light-weight. In that case, someone (maybe me) > could always write a REST interface, but it would merely consume the first-class > QMF API. I want to avoid having multiple, parallel APIs to maintain. If we like, with the RSDL added, I think I could write a bridge that would give you REST & RSDL automatically from QMF. Carl. From cctrieloff at redhat.com Wed Nov 30 22:00:33 2011 From: cctrieloff at redhat.com (Carl Trieloff) Date: Wed, 30 Nov 2011 17:00:33 -0500 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <20111130155237.GJ13803@us.ibm.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64C99.50706@redhat.com> <20111130155237.GJ13803@us.ibm.com> Message-ID: <4ED6A781.6010908@redhat.com> On 11/30/2011 10:52 AM, Adam Litke wrote: >> I'm not sure how hard it is technically. But for ISV's, I can tell >> > you that almost nobody has experience with it. > That is not a good sign. Certainly there must be some sort of standardized and > well understood API transport that we can use. We're not doing anything > particularly novel as far as the API is concerned There are a bunch of ISV's using it. Doc still lacking in some places but used in massive deployments. Carl. From agl at us.ibm.com Wed Nov 30 22:40:41 2011 From: agl at us.ibm.com (Adam Litke) Date: Wed, 30 Nov 2011 16:40:41 -0600 Subject: [Engine-devel] Proposed next-generation vdsm API Message-ID: <20111130224041.GM13803@us.ibm.com> Recently we've had some very productive discussions concerning the VDSM API. I want to attempt to refocus the discussion around an emerging proposal and see if we can agree on a sensible path forward. Based on the discussion, I have identified the following requirements that a new API for vdsm should have: 1.) Single API that can be consumed by ovirt-engine and ISVs - We don't want to maintain multiple parallel APIs - To develop a vendor ecosystem, we must have a robust external API to vdsm 2.) Full vdsm capabilities are exposed without requiring ovirt-engine - ovirt components should be modular and independently useful - Some deployments might want to manage nodes without ovirt-engine 3.) Standardized protocol with low overhead - Required for widespread adoption 4.) Support for asynchronous tasks and events - Needed by ovirt-engine and other consumers Based on these requirements, the following proposal has started to emerge: Create a REST API that will provide all of the functionality that is currently available via the xmlrpc interface (with the goal of deprecating xmlrpc once it becomes mature enough). To support advanced clustering features that ovirt-engine is planning, we'll write an QMF broker that can proxy the REST API onto a message bus. ovirt-engine will interact with vdsm exclusively over this bus but the REST API will be the principle API and the entry point for ISV apps. A REST API provides a light-weight and standard way to access all of the vdsm functionality. The REST API will handle events by exposing a new 'events' collection at the api root. REST users will use some sort of polling to collect these events. The details of this interface are being worked on. Several ways for minimizing the impact of polling have been discussed. The QMF broker can expose a publish/subscribe model for events as appropriate. Is this model an acceptable way to improve the vdsm API? I would like to hear the opinions of ovirt-engine developers, vdsm developers, and other stakeholders. Thanks for providing feedback on this proposal! -- Adam Litke IBM Linux Technology Center From gjansen at redhat.com Wed Nov 30 22:43:00 2011 From: gjansen at redhat.com (Geert Jansen) Date: Wed, 30 Nov 2011 23:43:00 +0100 Subject: [Engine-devel] Announcing a proof of concept REST API for VDSM In-Reply-To: <4ED6A781.6010908@redhat.com> References: <20111111220132.GA2726@us.ibm.com> <20111129193908.GA16747@x200.localdomain> <20111129200601.GD13803@us.ibm.com> <20111129205444.GC13696@sequoia.sous-sol.org> <20111129213604.GE13803@us.ibm.com> <4ED5D7C8.4090001@redhat.com> <20111130150024.GF13803@us.ibm.com> <4ED64727.1010107@redhat.com> <4ED64C99.50706@redhat.com> <20111130155237.GJ13803@us.ibm.com> <4ED6A781.6010908@redhat.com> Message-ID: <4ED6B174.5010501@redhat.com> On 11/30/2011 11:00 PM, Carl Trieloff wrote: > On 11/30/2011 10:52 AM, Adam Litke wrote: >>> I'm not sure how hard it is technically. But for ISV's, I can tell >>>> you that almost nobody has experience with it. >> That is not a good sign. Certainly there must be some sort of standardized and >> well understood API transport that we can use. We're not doing anything >> particularly novel as far as the API is concerned That standard is called HTTP :) > There are a bunch of ISV's using it. Doc still lacking in some places > but used in massive deployments. Can you give an examples outside the FSI? And also there's a huge difference between "using it" versus "creating an API for public consumption on top of it". In virtualization, all our competitors APIs are HTTP based (be it SOAP, REST, XML-RPC...) This includes VMware, Microsoft and Citrix. I don't know of any cloud API either that uses something else than HTTP. AMQP may give you a nice bus interface that is helpful as an internal building block to create a distributed application. But I do not see any use of it outside a very small niche. Therefore i do not believe that it is suitable as a transport for an API if that API is for public consumption. Regards, Geert From gjansen at redhat.com Wed Nov 30 23:37:00 2011 From: gjansen at redhat.com (Geert Jansen) Date: Thu, 01 Dec 2011 00:37:00 +0100 Subject: [Engine-devel] Proposed next-generation vdsm API In-Reply-To: <20111130224041.GM13803@us.ibm.com> References: <20111130224041.GM13803@us.ibm.com> Message-ID: <4ED6BE1C.8020007@redhat.com> Hi, i think this makes sense, but i'm not a VDSM expert. I did want to point out one other point, below: On 11/30/2011 11:40 PM, Adam Litke wrote: > Recently we've had some very productive discussions concerning the VDSM API. I > want to attempt to refocus the discussion around an emerging proposal and see if > we can agree on a sensible path forward. > > Based on the discussion, I have identified the following requirements that > a new API for vdsm should have: > > 1.) Single API that can be consumed by ovirt-engine and ISVs > - We don't want to maintain multiple parallel APIs > - To develop a vendor ecosystem, we must have a robust external API to > vdsm I have doubts around how useful the VDSM API will be for creating an ecosystem. If you look at most virtualization ISVs today, they want to integrate with a multi-node API and not a single-node API. The only use case that i know where integrating with a single node API is requested is when you're basically creating a virtualization management platform like oVirt itself. [Since we haven't met before, a brief intro... I have been responsible at Red Hat for buiding our virtualization ecosystem for the past year or so.] Regards, Geert