Moving Jenkins master ASAP

[Ewoud Kohl van Wijngaarden]

On Tue, Aug 07, 2012 at 07:25:07PM +0300, Itamar Heim wrote:
> On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
> >On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote:
> >>On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
> >>>We need to pick a new hosting solution for jenkins.ovirt.org.
> >>Itamar gave us this more direct list of what we need for hosting:
> >>
> >>We currently have:
> >>- - Jenkins master
> >>- - Gerrit
> >>- - 2 fc17 jenkins slaves
> >>- - 2 rhel 6 jenkins slaves
> >>
> >>"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
> >>
> >>"For jenkins master - quad core, 8-16GB RAM, a few disks for IO"
> >ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G
> >of ram and non raided sata disks.  The master is centos 6.3 and the
> >slave is fedora 17.  Between the two they are handling all the same
> >build jobs as ovirt.org in about half the time.  The reason why is 2
> >fold.  1) the signal sata drive is much faster then the EC2 storage so
> >the IO waits are limited on ovirt.info comparied to ovirt.org.  2 My
> >master and slave are on the same gigabit network. So moving around files
> >are 10x to 100x faster then ovirt.org.

I'd rather have a master who runs no jobs itself. This also means it can
be much lighter in terms of CPU and RAM. I expect the typical load on
drives will also be easier since it's more sequential, but this is
mostly speculation.

It's also better from a security point of view since you run less
untrusted code and it's much easier to reinstall a slave than a master.
Obvious downside is more network traffic.

> >>
> >>"Jenkins slaves - depends on pricing i guess, but we need quite a few
> >>of these to run the jobs."
> >>
> >>Are minimum need is:
> >>
> >>* A new bare metal host for Jenkins master.
> >>* A new bare metal or VM host for Gerrit.
> >>* 4 VMs for Jenkins slaves
> >Ovirt.info tell me that a solid master with 16G of ram should be able to
> >run 6 jobs and still feed several slaves so you could drop the 2 RHEL
> >6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to
> >replace the small Linode instance.
> >>
> >>I'm going to see what we can do with ~$150/mon, with the idea that we
> >>could replace the relatively small Linode instance we're spending
> >>~$40/mon on currently.
> >>
> >>A crazy idea - should we move the master to jenkins.ovirt.info right
> >>away to get the performance boost, as a stop-gap until we get a
> >>dedicated host in a datacenter? (Just trying to make us not dependent
> >>on Robert's house as if it's a datacenter.)
> >I am open to this.  It is my home so there is no backup generator or isp
> >but my connection is 70/35M so it should be able to support jenkins
> >needs.  I have already given eedri root access to jenkins.ovirt.info
> >incase that is a question in anyone mind.
>
> a home based service for jenkins is a bit troubling for me.
> but maybe we can share some of the load - say, try to run on it some
> jobs at patch level and provide feedback on gerrit?

I agree with your concerns on the home based service for something this
critical as a master. One solution could be to revamp the slave I
provide into a master and use Roberts infra as slaves. It's a VM running
on our production RHEV cluster and has sufficient HA.