Moving Jenkins master ASAP

[Robert Middleswarth]

On 08/07/2012 12:40 PM, Ewoud Kohl van Wijngaarden wrote:
> On Tue, Aug 07, 2012 at 07:25:07PM +0300, Itamar Heim wrote:
>> On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
>>> On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote:
>>>> On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
>>>>> We need to pick a new hosting solution for jenkins.ovirt.org.
>>>> Itamar gave us this more direct list of what we need for hosting:
>>>>
>>>> We currently have:
>>>> - - Jenkins master
>>>> - - Gerrit
>>>> - - 2 fc17 jenkins slaves
>>>> - - 2 rhel 6 jenkins slaves
>>>>
>>>> "For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
>>>>
>>>> "For jenkins master - quad core, 8-16GB RAM, a few disks for IO"
>>> ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G
>>> of ram and non raided sata disks.  The master is centos 6.3 and the
>>> slave is fedora 17.  Between the two they are handling all the same
>>> build jobs as ovirt.org in about half the time.  The reason why is 2
>>> fold.  1) the signal sata drive is much faster then the EC2 storage so
>>> the IO waits are limited on ovirt.info comparied to ovirt.org.  2 My
>>> master and slave are on the same gigabit network. So moving around files
>>> are 10x to 100x faster then ovirt.org.
> I'd rather have a master who runs no jobs itself. This also means it can
> be much lighter in terms of CPU and RAM. I expect the typical load on
> drives will also be easier since it's more sequential, but this is
> mostly speculation.
>
> It's also better from a security point of view since you run less
> untrusted code and it's much easier to reinstall a slave than a master.
> Obvious downside is more network traffic.
>
>>>> "Jenkins slaves - depends on pricing i guess, but we need quite a few
>>>> of these to run the jobs."
>>>>
>>>> Are minimum need is:
>>>>
>>>> * A new bare metal host for Jenkins master.
>>>> * A new bare metal or VM host for Gerrit.
>>>> * 4 VMs for Jenkins slaves
>>> Ovirt.info tell me that a solid master with 16G of ram should be able to
>>> run 6 jobs and still feed several slaves so you could drop the 2 RHEL
>>> 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to
>>> replace the small Linode instance.
>>>> I'm going to see what we can do with ~$150/mon, with the idea that we
>>>> could replace the relatively small Linode instance we're spending
>>>> ~$40/mon on currently.
>>>>
>>>> A crazy idea - should we move the master to jenkins.ovirt.info right
>>>> away to get the performance boost, as a stop-gap until we get a
>>>> dedicated host in a datacenter? (Just trying to make us not dependent
>>>> on Robert's house as if it's a datacenter.)
>>> I am open to this.  It is my home so there is no backup generator or isp
>>> but my connection is 70/35M so it should be able to support jenkins
>>> needs.  I have already given eedri root access to jenkins.ovirt.info
>>> incase that is a question in anyone mind.
>> a home based service for jenkins is a bit troubling for me.
>> but maybe we can share some of the load - say, try to run on it some
>> jobs at patch level and provide feedback on gerrit?
> I agree with your concerns on the home based service for something this
> critical as a master. One solution could be to revamp the slave I
> provide into a master and use Roberts infra as slaves. It's a VM running
> on our production RHEV cluster and has sufficient HA.
My concern with that is the current slave seems to have slow IO.  I 
don't really want to replace 1 slow IO monster with another slow IO 
monster.  Trust me I understand the desire to not run the master on 
something not in a proper DC.  I think Itamar Heim idea of dividing the 
load between the boxes makes since.  We can keep the important builds on 
ovirt.org and move the per patch builds to ovirt.info until we can build 
a proper Jenkins cluster.
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra


-- 
Thanks
Robert Middleswarth
@rmiddle (twitter/IRC)