Puppet design

[Ewoud Kohl van Wijngaarden]

On Wed, May 22, 2013 at 06:48:34PM +0200, Ewoud Kohl van Wijngaarden wrote:
> On Wed, May 22, 2013 at 05:40:46PM +0200, Ewoud Kohl van Wijngaarden wrote:
> > I added the EL6 hosts to our foreman instance, but Fedora has a newer
> > puppet version (2.7) than EPEL (2.6). Since a 2.7 client can't talk to a
> > 2.6 master, I'd like to propose we start using the puppetlabs repo and
> > upgrade to 3.1.

I upgraded all EL6 hosts we already added to puppet to 3.2. Fedora
clients are 2.7.

> This should have been a bit longer. Remember that anything we now decide
> can be changed later. I'm just giving many options.

Since there hasn't been a reply, I've started moving forward and
slightly updated the ovirt-infra-puppet repo on my github and pushed it
to foreman.ovirt.org. Currently it just ensures vim is installed and
quaid, eedri and I have an account.

> ## Repository location
>
> Currently there's a repository at ssh://puppet@foreman.ovirt.org/puppet
> with a post-receive hook installed that converts each branch into an
> environment in /etc/puppet/environments, but I think want to be on
> gerrit.ovirt.org so we need request a repository and figure out how to
> do deployments on merges. This could be a jenkins job, but maybe setting
> up gerrit to mirror is sufficient.

So far I've used the above repository, but I'd like to request a gerrit
repository for this.

> ## General repository layout
>
> I think we all agree we want to somehow manage it in a git repository.
> But when it comes to the modules, we have some options
>
> - Have modules as git submodules
> This is something we do at $employer and works reasonable well, you have
> good control over your modules. On the other hand, manual dependency
> checking. I heard that you can use puppet module to locally install and
> then check into git as well, but no experience here. Also, git
> submodules are a bit cumbersome to handle.
>
> - Use librarian-puppet
> The tagline on http://librarian-puppet.com/ is actually 'You can all
> stop using git submodules now'. I have no experience with it and would
> require modifications on the git hook if we decide to continue using
> that. One thing I am unsure of here is dependencies. When does
> librarian-puppet decide to update and will that break other modules?
>
> librarian-puppet seems like the cool choice, but I have no experience
> with it so can't judge it.

Given I had no experience and my repo was already set up using git
submodules, I chose to expand on that for now.

> ## Actual modules
>
> When we get to the actual modules, I generally like to use all the
> foreman installer modules to keep the systems in line, but one downside
> is already that theforeman/apache is quite different from
> puppetlabs/apache and both claim the name apache. While foreman is
> moving towards more common modules, that's something I expect for 1.3
> (though we can just run development modules when that becomes an
> option).
>
> Then also comes the question of which modules we want to use and what we
> actually want to automate with puppet. I'd suggest to cheat here by
> making it up as we go along. Just set a goal (manage user logins for
> example) and just grow from there.

Right now I've only written a small module named ovirt_infra and added
puppet-jenkins + puppet-stdlib, but only use ovirt_infra. We will start
using more modules.

> ## Data
>
> Assuming we have a puppet repository, some modules with logic, we will
> need to add data. Again, multiple options.
>
> - Have a data module
> Just statically write all the data in a separate data module. Not that
> desirable I think, but certainly an option
>
> - Set parameters through foreman
> Foreman can override class parameters through the ENC, but you can't add
> defines here. So you may end up with a mix of data module and data in
> another system.
>
> - Hiera
> Since puppet 3.0 fully integrated, allows you to override class
> parameters in a hierarchical manner using multiple backends. Also allows
> you to add defines. One big benefit can be that you can manage it in a
> separate git repositories due to the hierarchical nature. One public and
> one private. The former can be for public data and the latter can also
> hold SSL private keys, maybe passwords. One challenge may be to keep
> multiple repositories in sync in terms of deployment.

I started expanding the hardcoded option, but soon ran into the password
hash problem so that's already being set in foreman. After a short
discussion with David, we were strongly considering hiera but the
current solution works for now.

> ## Concrete proposal
>
> A long time ago I wrote https://github.com/ekohl/ovirt-infra-puppet.
> It's set up using the submodules way, only has a single module (for
> jenkins) and hardcodes data, but it is a start. It has a decent start to
> add users and I'd like to expand on that. That way we can easily create
> accounts for us admins on all servers.

So I've started doing this, but we could add the rest of the admins. We
should also

> The next step could be to add puppetdb to our setup so we can use
> exported resources to generate the icinga hosts list, but an easier
> option is to start setting up a resources.ovirt.org configuration so we
> can start to migrate linode01. resources.ovirt.org is just serving
> static content so should be very easy.

I'd like to work with Ohad to set up nrpe on all hosts and get some
monitoring. Initially statically, but maybe it's just as easy to set up
exported resources and generate the hosts configuration for icinga.