Proposing a new infra design

[Eyal Edri]

----- Original Message -----
> From: "Kiril Nesenko" <kiril at redhat.com>
> To: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt at kohlvanwijngaarden.nl>
> Cc: infra at ovirt.org
> Sent: Sunday, June 9, 2013 3:03:32 PM
> Subject: Re: Proposing a new infra design
> 
> 
> ----- Original Message -----
> > From: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt at kohlvanwijngaarden.nl>
> > To: infra at ovirt.org
> > Sent: Sunday, June 9, 2013 1:33:54 PM
> > Subject: Re: Proposing a new infra design
> > 
> > On Sun, Jun 09, 2013 at 02:40:02AM -0400, Kiril Nesenko wrote:
> > > I would like to propose a new design for our infrastructure.
> > 
> > First of all, I'm happy to revisit this. Inline are some comments that
> > explain why some choices were made and some thoughts.
> > 
> > > Today we have some bare metal hosts:
> > >
> > > USA
> > > rackspace01.ovirt.org - f18 clean
> > > rackspace02.ovirt.org - f18 clean
> > >
> > >
> > > France
> > > alterway01.ovirt.org - jenkins.ovirt.org
> > > alterway02.ovirt.org - all-in-one (that run few our services like
> > > puppet,foreman etc.)
> > >
> > > I think that it make no sense to install all-in-one setups with local
> > > storage domains on our bare metal hosts.  It will be better to install
> > > NFS or ISCSI DCs per location. For example we can use amazon VMs as
> > > engines and bare metal hosts as hypervisors.
> > >
> > > This is the design that I propose:
> > >
> > > amazon vm1(engine) USA -
> > >                    |
> > >                    |- rackspace01.ovirt.org
> > >                    |- rackspace02.ovirt.org
> > >
> > > amazon vm2(engine) France -
> > >                    |
> > >                    |- alterway01.ovirt.org
> > >                    |- alterway02.ovirt.org
> > 
> > I do like the idea of one more than one host per ovirt environment. HA
> > is really something we should have for some services.
> > 
> > The reason we installed jenkins on a phyiscal host was partly based on
> > that we just needed a new jenkins master and had little time to make our
> > optimal environment. We also said that we could re-evaluate that choice
> > at a later time.
> > 
> > Regarding the amazon engines: don't you need some layer 2 networking
> > between the manager and the hosts? Wouldn't a VM at rackspace be much
> > better as an engine because of lower latency? Maybe rackspace can just
> > provide a VLAN between the engine and hosts which should make our
> > management much easier? Maybe we can try to achieve a similar situation
> > at Alterway?
> 
> If would be great !
> 
> > 
> > > - Storage
> > > * For this design we need storage services that will be located in the
> > >   same DCs as our bare metal hosts.
> > 
> > Could we use gluster where possible? At alterway for example. For
> > rackspace I'd prefer local storage per node, but I'll get to that later.
> 
> gluster is a possible solution, but for gluster we still need external
> storage.
> Why do you want to use a local storage :) ?
> 
> > 
> > > * Also we will need a storage for our backups - today we don't have a
> > >   defined location for our backups.
> > 
> > Amazon S3 maybe?
> 
> Possible. We just need a place where we can save our backups.
> 
> > 
> > > * Storage for resources.ovirt.org - make no sense that VM stores RPMs
> > >   on it. Much better to use a VM with a small HD and use external
> > >   storage for storing RPMs.
> > 
> > I don't quite understand this. I get that you'd want different
> > partitions, but why external storage? Whether the host manages this or
> > the guest, does it really make a difference?
> > 
> > When we get our environment up, I'd like to look into setting up mirrors
> > where people/orgs can sponsor some mirror so resources.ovirt.org becomes
> > less important anyway. yum has a nice mirrorlist feature and I'm sure we
> > can work something out.
> 
> I am not sure on which servers resources.ovirt.org is running right now, but
> I would
> like to run our infra on our servers. For this purpose its better to create a
> VMs with a small HD and use ext. storage
> to save RPMs on it.

Just something we heard of on last jenkins conference - https://bintray.com/bintray/rpm-center
might worth checking as a replacement for ovirt rpms repository.

it has REST API [1], so we might be able to publish rpms from jenkins directly to it.

[1] https://bintray.com/docs/rest/api.html


> 
> > 
> > > All services/Vms from alterway02 will be exported to USA DC using NFS
> > > export domain.
> > > USA DC will be used for production machines.
> > > France DC will be used for jenkins slaves.
> > 
> > One thing you're forgetting is that the rackspace machines are faster.
> > The fast local storage (SSD on one, 10k RPM SAS on the other IIRC),
> > newer CPUs with more cores and more RAM make them much better candidates
> > for jenkins slaves. That's why I think we should them as hosts for
> > jenkins slaves with local storage for the slaves on rackspace.
> > 
> > That also means we should keep using alterway as our production
> > environemnt but because of that I'd really like a pair of ovirt hosts to
> > make it HA. In these times we shouldn't have to announce downtime
> > because we need to upgrade the kernel on one of our hosts for example.
> 
> I wasn't aware of rackspace specs. Sounds good to me :).
> 
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> > 
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>