Moving Jenkins master ASAP

[Eyal Edri]

----- Original Message -----
> From: "Robert Middleswarth" <robert at middleswarth.net>
> To: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt at kohlvanwijngaarden.nl>
> Cc: infra at ovirt.org
> Sent: Wednesday, August 1, 2012 6:02:06 AM
> Subject: Re: Moving Jenkins master ASAP
> 
> On 07/31/2012 07:12 PM, Ewoud Kohl van Wijngaarden wrote:
> > On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth
> > wrote:
> >> On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
> >>> On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade
> >>> wrote:
> >>>> On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
> >>>>> We need to pick a new hosting solution for jenkins.ovirt.org.
> >>>>>
> >>>>> One idea is for us to throw out some favorite hosting providers
> >>>>> here, and see if we can sort out what would be a good solution.
> >>>> This post is what made me aware that EC2 would be a dead-end for
> >>>> us
> >>>> for now:
> >>>>
> >>>> http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
> >>>>
> >>>> In that post, the author used this host for comparison testing:
> >>>>
> >>>> http://joesdatacenter.com/
> >>> My employer is a hosting provider so I'm somewhat biased here.
> >> It not just about the provider.  I would need to see the bandwidth
> >> charts on the current Jenkins but I assume Just about any provider
> >> can handle it bandwidth needs.  But the server Jenkins Master
> >> needs
> >> to run on.  EC2 isn't cutting it.  My testing box is a basic Sata
> >> drive and it is running much faster but there is no user load on
> >> the
> >> box.  We really need a box with raid 10 drives in it to handle the
> >> high IO needs.
> > http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html
> > are
> > the stats of the jenkins slave we (my employer) provide. This is a
> > production load. Quick analysis shows that IO is limiting at times,
> > but
> > the high IO peaks correlate to the swap. So adding more than 8GB
> > RAM
> > would lessen the requirement on the IO. Note that it is currently
> > running on our SATA SAN, but I don't know the RAID config from the
> > top
> > of my head.
> Slave boxes are diff from the master.  Jenkins copies all the files
> over
> from the master to the slave then back up to the master.  Using a
> good
> chunk of bandwidith and disk IO on both the slaves and the master.
> Every job requires IO on the master and a lot of it.  As the number
> of
> slaves goes up so does the IO on the master.  The current EC2
> instance
> isn't holding it own with load.  Spikes can literately take it
> offline
> and even when it is idle it still is showing a ton of IO from the
> people
> visiting the site.  The question is with the limited budget what can
> we do.
> 
> What we really want for the master is a dedicated machine with a sas
> /
> ssd raid 10 controller aka the profile of a database server.  What we
> can get away with for now is the real question.  I can offer up my
> boxes
> they are just running on Sata drives and currently running behind my
> 70/35 Verizon FiOS connection ( Jenkins.ovirt.info ).  We could move
> them to a local co-lo for about $40.00 a U per month a friend of mine
> runs.
> 
> What does everyone else think?

we can look at jenkins master load on jenkins.ovirt.org/monitoring (you need to be jenkins administrator to see it). 

> 
> --
> Thanks
> Robert Middleswarth
> @rmiddle (twitter/IRC)
> 
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>