Proposing a new infra design

[Ewoud Kohl van Wijngaarden]

On Sun, Jun 09, 2013 at 02:40:02AM -0400, Kiril Nesenko wrote:
> I would like to propose a new design for our infrastructure.

First of all, I'm happy to revisit this. Inline are some comments that
explain why some choices were made and some thoughts.

> Today we have some bare metal hosts:
>
> USA
> rackspace01.ovirt.org - f18 clean
> rackspace02.ovirt.org - f18 clean
>
>
> France
> alterway01.ovirt.org - jenkins.ovirt.org
> alterway02.ovirt.org - all-in-one (that run few our services like puppet,foreman etc.)
>
> I think that it make no sense to install all-in-one setups with local
> storage domains on our bare metal hosts.  It will be better to install
> NFS or ISCSI DCs per location. For example we can use amazon VMs as
> engines and bare metal hosts as hypervisors.
>
> This is the design that I propose:
>
> amazon vm1(engine) USA -
>                    |
>                    |- rackspace01.ovirt.org
>                    |- rackspace02.ovirt.org
>
> amazon vm2(engine) France -
>                    |
>                    |- alterway01.ovirt.org
>                    |- alterway02.ovirt.org

I do like the idea of one more than one host per ovirt environment. HA
is really something we should have for some services.

The reason we installed jenkins on a phyiscal host was partly based on
that we just needed a new jenkins master and had little time to make our
optimal environment. We also said that we could re-evaluate that choice
at a later time.

Regarding the amazon engines: don't you need some layer 2 networking
between the manager and the hosts? Wouldn't a VM at rackspace be much
better as an engine because of lower latency? Maybe rackspace can just
provide a VLAN between the engine and hosts which should make our
management much easier? Maybe we can try to achieve a similar situation
at Alterway?

> - Storage
> * For this design we need storage services that will be located in the
>   same DCs as our bare metal hosts.

Could we use gluster where possible? At alterway for example. For
rackspace I'd prefer local storage per node, but I'll get to that later.

> * Also we will need a storage for our backups - today we don't have a
>   defined location for our backups.

Amazon S3 maybe?

> * Storage for resources.ovirt.org - make no sense that VM stores RPMs
>   on it. Much better to use a VM with a small HD and use external
>   storage for storing RPMs.

I don't quite understand this. I get that you'd want different
partitions, but why external storage? Whether the host manages this or
the guest, does it really make a difference?

When we get our environment up, I'd like to look into setting up mirrors
where people/orgs can sponsor some mirror so resources.ovirt.org becomes
less important anyway. yum has a nice mirrorlist feature and I'm sure we
can work something out.

> All services/Vms from alterway02 will be exported to USA DC using NFS export domain.
> USA DC will be used for production machines.
> France DC will be used for jenkins slaves.

One thing you're forgetting is that the rackspace machines are faster.
The fast local storage (SSD on one, 10k RPM SAS on the other IIRC),
newer CPUs with more cores and more RAM make them much better candidates
for jenkins slaves. That's why I think we should them as hosts for
jenkins slaves with local storage for the slaves on rackspace.

That also means we should keep using alterway as our production
environemnt but because of that I'd really like a pair of ovirt hosts to
make it HA. In these times we shouldn't have to announce downtime
because we need to upgrade the kernel on one of our hosts for example.