Re: Moving Jenkins master ASAP

----- Original Message ----- > From: "Robert Middleswarth" <robert(a)middleswarth.net&gt; > To: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt(a)kohlvanwijngaarden.nl&gt; > Cc: infra(a)ovirt.org > Sent: Wednesday, August 1, 2012 6:02:06 AM > Subject: Re: Moving Jenkins master ASAP > > On 07/31/2012 07:12 PM, Ewoud Kohl van Wijngaarden wrote: >> On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth >> wrote: >>> On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote: >>>> On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade >>>> wrote: >>>>> On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote: >>>>>> We need to pick a new hosting solution for jenkins.ovirt.org. >>>>>> >>>>>> One idea is for us to throw out some favorite hosting providers >>>>>> here, and see if we can sort out what would be a good solution. >>>>> This post is what made me aware that EC2 would be a dead-end for >>>>> us >>>>> for now: >>>>> >>>>> http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/ >>>>> >>>>> In that post, the author used this host for comparison testing: >>>>> >>>>> http://joesdatacenter.com/ >>>> My employer is a hosting provider so I'm somewhat biased here. >>> It not just about the provider. I would need to see the bandwidth >>> charts on the current Jenkins but I assume Just about any provider >>> can handle it bandwidth needs. But the server Jenkins Master >>> needs >>> to run on. EC2 isn't cutting it. My testing box is a basic Sata >>> drive and it is running much faster but there is no user load on >>> the >>> box. We really need a box with raid 10 drives in it to handle the >>> high IO needs. >> http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html >> are >> the stats of the jenkins slave we (my employer) provide. This is a >> production load. Quick analysis shows that IO is limiting at times, >> but >> the high IO peaks correlate to the swap. So adding more than 8GB >> RAM >> would lessen the requirement on the IO. Note that it is currently >> running on our SATA SAN, but I don't know the RAID config from the >> top >> of my head. > Slave boxes are diff from the master. Jenkins copies all the files > over > from the master to the slave then back up to the master. Using a > good > chunk of bandwidith and disk IO on both the slaves and the master. > Every job requires IO on the master and a lot of it. As the number > of > slaves goes up so does the IO on the master. The current EC2 > instance > isn't holding it own with load. Spikes can literately take it > offline > and even when it is idle it still is showing a ton of IO from the > people > visiting the site. The question is with the limited budget what can > we do. > > What we really want for the master is a dedicated machine with a sas > / > ssd raid 10 controller aka the profile of a database server. What we > can get away with for now is the real question. I can offer up my > boxes > they are just running on Sata drives and currently running behind my > 70/35 Verizon FiOS connection ( Jenkins.ovirt.info ). We could move > them to a local co-lo for about $40.00 a U per month a friend of mine > runs. > > What does everyone else think? we can look at jenkins master load on jenkins.ovirt.org/monitoring (you need to be jenkins administrator to see it).