[Engine-devel] [ANN] New development environment for ovirt-engine

Tue May 14 13:28:05 UTC 2013

On Tue, May 14, 2013 at 1:54 PM, Moti Asayag <masayag at redhat.com> wrote:
> 
> 
> ----- Original Message -----
>>  From: "Alon Bar-Lev" <alonbl at redhat.com>
>>  To: "engine-devel" <engine-devel at ovirt.org>
>>  Cc: "Yaniv Bronheim" <ybronhei at redhat.com>, "Moti Asayag" 
>> <masayag at redhat.com>, "Limor Gavish" <lgavish at gmail.com>,
>>  "Sharad Mishra" <snmishra at us.ibm.com>, "Alex Lourie" 
>> <alourie at redhat.com>, "Sandro Bonazzola" <sbonazzo at redhat.com>,
>>  "arch" <arch at ovirt.org>, "Ofer Schreiber" <oschreib at redhat.com>
>>  Sent: Sunday, May 12, 2013 2:52:51 PM
>>  Subject: [ANN] New development environment for ovirt-engine
>>  
>>  Hello all ovirt-engine developers,
>>  
>>  When I first joined the ovirt project, it took me about two weeks 
>> to setup a
>>  development environment, I needed to work on a bug related to 
>> host-deploy so
>>  I needed an environment that could use the ssh, PKI, vdsm-bootstrap 
>> and
>>  communicate with vdsm using SSL, this was virtually impossible to 
>> do so
>>  without tweaking the product in a way that it is so different from
>>  production use, that I cannot guarantee that whatever tested in 
>> development
>>  will actually work in production.
>>  
>>  I peeked at the installation script in a hope that I can create 
>> partial
>>  environment similar to production, but I found that the packaging
>>  implementation makes to much assumption and is very difficult to 
>> adopt. The
>>  fact that I do not use fedora/rhel for my development made it even 
>> worse.
>>  
>>  I had no other option than to create rpms after each of my changes 
>> and test
>>  each in real production like setup.
>>  
>>  It was obvious to me that the manual customization of developers to 
>> achieve
>>  working product will eventually break as product grow and move away 
>> from
>>  being developer friendly to production friendly. For example, 
>> product
>>  defaults cannot be these which serve developers, but these which 
>> serve
>>  production the best, or having a valid PKI setup cannot be optional 
>> any more
>>  as components do need to use it. Same for location of files and
>>  configuration, for example, if we write a pluggable infrastructure 
>> for
>>  branding, we cannot damage the interface just because developers 
>> runs the
>>  product in their own manual customization.
>>  
>>  I took the opportunity handed to me to port the ovirt-engine to 
>> other
>>  distributions in order to provide a development environment that is 
>> similar
>>  to production setup. Together with Sandro Bonazzola and Alex Lourie 
>> we
>>  re-wrote the whole installation of the product which can also be 
>> used to
>>  setup the desired development environment.
>>  
>>  Within this environment the product is set up using the same tools 
>> and
>>  configuration as in production, while the process does not require 
>> special
>>  privileges nor changes the state of the developer machine.
>>  
>>  A complete documentation is available[1], I preferred to use README 
>> within
>>  the source tree as wiki tend to quickly become obsolete, while 
>> documentation
>>  within source tree can be modified by the commit that introduces a 
>> change. I
>>  will redirect to this file from the current wiki once the site will 
>> be up.
>>  
>>  In a nut shell, after installing prerequisites, build and install 
>> the product
>>  using:
>>  
>>  $ make clean install-dev PREFIX=$HOME/ovirt-engine
>>  
>>  This will run maven and create product installation at 
>> $HOME/ovirt-engine
>>  Next, a setup phase is required just like in production, to 
>> initialize
>>  configuration and database:
>>  
>>  $ $HOME/ovirt-engine/bin/engine-setup-2
>>  
>>  You have now fully functional product, including PKI, SSL, 
>> host-deploy,
>>  tools.
>>  No manual database updates are required, no lose of functionality.
>>  
>>  All that is left is to start the engine service:
>>  
>>  $ $HOME/ovirt-engine/share/ovirt-engine/services/ovirt-engine.py 
>> start
>>  
>>  Access to application:
>>     http://localhost:8080
>>     https://localhost:8443
>>  Debugging port is opened at port 8787.
>>  
>>  Farther information exists in the documentation[1].
>>  
>>  There are several inherit benefits of the new environment, the 
>> major one is
>>  the ability to manage several environments in parallel on the same 
>> host. For
>>  example, if we develop two separate features on two branches we can 
>> install
>>  the product into $HOME/ovirt-engine-feature1 and
>>  $HOME/ovirt-engine-feature-2 and have a separate database for each, 
>> if we
>>  modify the ports jboss is listening to we can run two instances of 
>> engine at
>>  the same time!
>>  
>>  We will be happy to work with all developers to assist in porting 
>> into the
>>  new development environment, the simplest is to create a new 
>> database for
>>  this effort. Moti has a sequence of converting the existing 
>> database owned
>>  by postgres to be owned by the engine, Moti, can you please share 
>> that?
>>  
>> 
> Reusing an existing DB schema requires a bit more work since the 
> installation 
> of dev-env advise using database as a user and not a superuser like 
> 'postgres'
> user used to create the database originally. 
> 

We no longer (since 3.1) create the DB in production with user 
postgres. All the DB operations are done with application user (engine 
by default), especially with remote DB installations. The application 
user doesn't have superuser privileges, only the ones required for DB 
creation and removal.

> 
> Therefore if wishes to use user 'engine' as in the instructions, 
> there is a need
> to change the current schema owner and also the ownership of all of 
> its objects.
> 
> The easiest path I found for that purpose is:
> 1. Create a dump of the database using the script in [1].
> 2. Rename the owner in the dump file to the new owner (s/OWNER TO 
> postgres/OWNER TO engine/g).
> 3. Import the dump file to the new DB owned by the engine user using 
> [2] (provide -r flag to
> drop the former db).
> 
> [1] ovirt-engine/backend/manager/dbscripts/backup.sh
> [2] ovirt-engine/backend/manager/dbscripts/restore.sh
> 
>>  We are sure there are missing bits, we will be happy to know these 
>> so we can
>>  improve.
>>  
>>  I am aware that developers (especially java) are conservative, but 
>> I ask you
>>  to give us a chance, so that we make it easy for developers to join 
>> the
>>  project, and to allow us to drop the parallel effort of packaging to
>>  production and fixing the broken development environment.
>>  
>>  A special thanks to developers who took the time to test and 
>> provide feedback
>>  before the merged:
>>  - Yaniv Bronheim
>>  - Moti Asayag
>>  - Limor Gavish
>>  - Sharad Mishra
>>  - Ofer Schreiber
>>  
>>  We are hoping that after migration you will be find this 
>> environment useful
>>  and friendly,
>>  
>>  Sandro Bonazzola,
>>  Alex Lourie,
>>  Alon Bar-Lev.
>>  
>>  [1]
>>  
>> http://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=README.developer;hb=HEAD
>>  
>> 