Puppet modules organization

[David Caro]

On Tue 17 Jun 2014 04:54:45 PM CEST, Ewoud Kohl van Wijngaarden wrote:
> On Tue, Jun 17, 2014 at 11:36:35AM +0200, David Caro wrote:
>> Hi everyone!
>>
>> I'm starting a thread to discuss the puppet modules organization.
>>
>> There are two proposed ways of organizing them:
>>
>> 1.- Using a unique module named ovirt_infra
>> 2.- Using multiple modules, named ovirt_*
>>
>> Feel free to propose other alternatives, the main points for each one are:
>>
>> 1.- Everything inside one module, easy to find
>> 1.- Easy to add a new class, just create the file
>> 1.- Easy to create hard to maintain code
>> 1.- Easy to create very interdependent code
>>
>>
>> 2.- Enforces modularization of the different code (one module, one task), that
>> brings
>> 2.- Easier to test
>> 2.- Safe to reuse
>> 2.- More organized (not everything in the same place)
>> 2.- It's the most common way of organizing puppet manifests, so the main
>> guidelines, patterns and most of the documentation expects this way of working.
>>
>>
>> Please send your comments and if too many I'll open a pad with the them for easy
>> review.
>>
>>
>> I vote for #2, modularized organization.
>
> I'd be in favor of a roles/profiles pattern[1]. Given we use foreman, I
> think the roles pattern is filled by foreman hosts (including host
> groups and since foreman 1.5 configuration groups). That leaves a
> profiles module. I think ovirt_infra suits that, so option 1.

That's not against the modules view I expressed before, and it's against what we
have now, let me explain:

A basic example, right now we have the class ovirt_infra::user, that is not a
profile, and it does not belong inside a profile module.

If you see the link that you sent [1] it has this structure:

Node -> Role -> Profiles -> Modules -> Resources

Where a profile is:
"Unlike roles, which are named in a more human representation of the server
function, a profile incorporates individual components to represent a logical
technology stack"

The users class does not fit in any case in that definition. Instead it's in the
modules level.

But right now, our modules level is empty (we just have external modules there)
and what I propose is to extract all the individual classes that do not fit the
profiles definition to their own module (one per class, as they should be
logically independent at that level).

Just as it explains on the same paragraph:
"the webserver profile is made up of the httpd, memcache and php components. In
Puppet, these lower level components are represented by your modules."

So we are not using the profiles/roles pattern right now. And I proposed using
it (maybe not that explicitly, but that was the idea), and that require the
modularization of a few classes that we already have.

That post also agrees with modularization:
"I’ve talked about profiles and roles like they are some special case and
modules being something else. In reality, all of these classes can be, and
should be modularised. I make a logical distinction between the profile and role
modules, and everything else (e.g.: modules that provide resources)."

>
> That said, I'm not against renaming ovirt_infra to profiles and adding a
> roles module. Then in foreman you only select a single role.

I vote for this also, so we will end up with a layout similar to this:

site
 | - role
 |    | - manifests
 |    |    | - puppetmaster.pp
 |    |    | - jenkins_master.pp
 |    |    | - jenkins_slave.pp
 |    |    ...
 | - profile
 |    | - manifests
 |    | - base.pp
 |    | - jnlp_slave.pp
 |    | - www.pp
 |    | - nagios.pp
 |    ...	
 | - ovirt_mod1 (custom module that is not a profile, like ovirt_user)
 | - ovirt_mod2 (custom module 2, for example, ovirt_repo)
...

That will help us easily control changes on the configuration of the hostgroups,
and review changes on that configuration too. But we will need another repo to
hold the sensitive parameters (passwords and all), maybe using hiera (I've been
playing with non-public gerrit repos, that can be useful).

>
> Also, I disagree it's the main guideline to build one module per role.
> If your task is to define all profiles, then it's still one task.
>
> Another disadvantage of many modules is that many of the standard puppet
> testing tools work per module. If you want to write tests for every
> module, you end up with a lot of files. We can have a look at writing
> custom scripts that perform all tests, but there's something to say
> about using standard tools puppet module authors are used to. (See
> garethr/puppet-module-skeleton[1] for some ideas of a standard module.)

I don't see this as a disadvantage, but as an advantage, having tests for each
module means that the tests are specific to that module, are clearer and easier
to write and troubleshoot. From that same author you can see that he's quite
used to create modules, instead of having one big module per installation.
It's fairly easy to write a script to run the tests on each module, that should
be no problem.

>
> [1]: http://www.craigdunn.org/2012/05/239/
> [2]: https://github.com/garethr/puppet-module-skeleton
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra


-- 
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Email: dcaro at redhat.com
Web: www.redhat.com
RHT Global #: 82-62605

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20140617/045cd576/attachment.sig>