Moving Jenkins master ASAP
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We need to pick a new hosting solution for jenkins.ovirt.org. One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution. Other ideas have been floated. Ideally, we'll get more hardware in the future via some hosting that Red Hat is working on providing for projects. In the meantime ... Do any other sponsoring organizations have resources we can look in to? Once we pick something, we can move it all fairly quickly, I think. Can we target the end of this week? Perhaps, if we don't run in to any complications ... - - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFQF+9b2ZIOBq0ODEERAgztAKCKGv3WgUlIpOQ3AvgnY7V+r42C0ACffdb7 CA0i4zoGWeWGS52n3tW/SLk= =xKs6 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution.
This post is what made me aware that EC2 would be a dead-end for us for now: http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/ In that post, the author used this host for comparison testing: http://joesdatacenter.com/ - - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFQF/Ep2ZIOBq0ODEERAtI+AJ4syPPz7jll19ypBy5IsjtcfqhNYQCfRKW2 Wvwb/Vd3wltWNnmOZD+OmLI= =G1Ud -----END PGP SIGNATURE-----
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution.
This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
My employer is a hosting provider so I'm somewhat biased here.
On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution. This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
http://joesdatacenter.com/ My employer is a hosting provider so I'm somewhat biased here.
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
It not just about the provider. I would need to see the bandwidth charts on the current Jenkins but I assume Just about any provider can handle it bandwidth needs. But the server Jenkins Master needs to run on. EC2 isn't cutting it. My testing box is a basic Sata drive and it is running much faster but there is no user load on the box. We really need a box with raid 10 drives in it to handle the high IO needs. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth wrote:
On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution. This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
http://joesdatacenter.com/ My employer is a hosting provider so I'm somewhat biased here.
It not just about the provider. I would need to see the bandwidth charts on the current Jenkins but I assume Just about any provider can handle it bandwidth needs. But the server Jenkins Master needs to run on. EC2 isn't cutting it. My testing box is a basic Sata drive and it is running much faster but there is no user load on the box. We really need a box with raid 10 drives in it to handle the high IO needs.
http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html are the stats of the jenkins slave we (my employer) provide. This is a production load. Quick analysis shows that IO is limiting at times, but the high IO peaks correlate to the swap. So adding more than 8GB RAM would lessen the requirement on the IO. Note that it is currently running on our SATA SAN, but I don't know the RAID config from the top of my head.
On 07/31/2012 07:12 PM, Ewoud Kohl van Wijngaarden wrote:
On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution. This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
http://joesdatacenter.com/ My employer is a hosting provider so I'm somewhat biased here. It not just about the provider. I would need to see the bandwidth charts on the current Jenkins but I assume Just about any provider can handle it bandwidth needs. But the server Jenkins Master needs to run on. EC2 isn't cutting it. My testing box is a basic Sata drive and it is running much faster but there is no user load on the box. We really need a box with raid 10 drives in it to handle the high IO needs. http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html are
On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth wrote: the stats of the jenkins slave we (my employer) provide. This is a production load. Quick analysis shows that IO is limiting at times, but the high IO peaks correlate to the swap. So adding more than 8GB RAM would lessen the requirement on the IO. Note that it is currently running on our SATA SAN, but I don't know the RAID config from the top of my head. Slave boxes are diff from the master. Jenkins copies all the files over from the master to the slave then back up to the master. Using a good chunk of bandwidith and disk IO on both the slaves and the master. Every job requires IO on the master and a lot of it. As the number of slaves goes up so does the IO on the master. The current EC2 instance isn't holding it own with load. Spikes can literately take it offline and even when it is idle it still is showing a ton of IO from the people visiting the site. The question is with the limited budget what can we do.
What we really want for the master is a dedicated machine with a sas / ssd raid 10 controller aka the profile of a database server. What we can get away with for now is the real question. I can offer up my boxes they are just running on Sata drives and currently running behind my 70/35 Verizon FiOS connection ( Jenkins.ovirt.info ). We could move them to a local co-lo for about $40.00 a U per month a friend of mine runs. What does everyone else think? -- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
----- Original Message -----
From: "Robert Middleswarth" <robert@middleswarth.net> To: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt@kohlvanwijngaarden.nl> Cc: infra@ovirt.org Sent: Wednesday, August 1, 2012 6:02:06 AM Subject: Re: Moving Jenkins master ASAP
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution. This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
http://joesdatacenter.com/ My employer is a hosting provider so I'm somewhat biased here. It not just about the provider. I would need to see the bandwidth charts on the current Jenkins but I assume Just about any provider can handle it bandwidth needs. But the server Jenkins Master needs to run on. EC2 isn't cutting it. My testing box is a basic Sata drive and it is running much faster but there is no user load on
On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote: the box. We really need a box with raid 10 drives in it to handle the high IO needs. http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html are
On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth wrote: the stats of the jenkins slave we (my employer) provide. This is a production load. Quick analysis shows that IO is limiting at times, but the high IO peaks correlate to the swap. So adding more than 8GB RAM would lessen the requirement on the IO. Note that it is currently running on our SATA SAN, but I don't know the RAID config from the top of my head. Slave boxes are diff from the master. Jenkins copies all the files over from the master to the slave then back up to the master. Using a good chunk of bandwidith and disk IO on both the slaves and the master. Every job requires IO on the master and a lot of it. As the number of slaves goes up so does the IO on the master. The current EC2 instance isn't holding it own with load. Spikes can literately take it offline and even when it is idle it still is showing a ton of IO from the
On 07/31/2012 07:12 PM, Ewoud Kohl van Wijngaarden wrote: people visiting the site. The question is with the limited budget what can we do.
What we really want for the master is a dedicated machine with a sas / ssd raid 10 controller aka the profile of a database server. What we can get away with for now is the real question. I can offer up my boxes they are just running on Sata drives and currently running behind my 70/35 Verizon FiOS connection ( Jenkins.ovirt.info ). We could move them to a local co-lo for about $40.00 a U per month a friend of mine runs.
What does everyone else think?
we can look at jenkins master load on jenkins.ovirt.org/monitoring (you need to be jenkins administrator to see it).
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
On 08/01/2012 07:52 AM, Eyal Edri wrote:
----- Original Message -----
From: "Robert Middleswarth" <robert@middleswarth.net> To: "Ewoud Kohl van Wijngaarden" <ewoud+ovirt@kohlvanwijngaarden.nl> Cc: infra@ovirt.org Sent: Wednesday, August 1, 2012 6:02:06 AM Subject: Re: Moving Jenkins master ASAP
On Tue, Jul 31, 2012 at 07:52:25AM -0700, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote: > We need to pick a new hosting solution for jenkins.ovirt.org. > > One idea is for us to throw out some favorite hosting providers > here, and see if we can sort out what would be a good solution. This post is what made me aware that EC2 would be a dead-end for us for now:
http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/
In that post, the author used this host for comparison testing:
http://joesdatacenter.com/ My employer is a hosting provider so I'm somewhat biased here. It not just about the provider. I would need to see the bandwidth charts on the current Jenkins but I assume Just about any provider can handle it bandwidth needs. But the server Jenkins Master needs to run on. EC2 isn't cutting it. My testing box is a basic Sata drive and it is running much faster but there is no user load on
On 07/31/2012 02:16 PM, Ewoud Kohl van Wijngaarden wrote: the box. We really need a box with raid 10 drives in it to handle the high IO needs. http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html are
On Tue, Jul 31, 2012 at 02:57:56PM -0400, Robert Middleswarth wrote: the stats of the jenkins slave we (my employer) provide. This is a production load. Quick analysis shows that IO is limiting at times, but the high IO peaks correlate to the swap. So adding more than 8GB RAM would lessen the requirement on the IO. Note that it is currently running on our SATA SAN, but I don't know the RAID config from the top of my head. Slave boxes are diff from the master. Jenkins copies all the files over from the master to the slave then back up to the master. Using a good chunk of bandwidith and disk IO on both the slaves and the master. Every job requires IO on the master and a lot of it. As the number of slaves goes up so does the IO on the master. The current EC2 instance isn't holding it own with load. Spikes can literately take it offline and even when it is idle it still is showing a ton of IO from the
On 07/31/2012 07:12 PM, Ewoud Kohl van Wijngaarden wrote: people visiting the site. The question is with the limited budget what can we do.
What we really want for the master is a dedicated machine with a sas / ssd raid 10 controller aka the profile of a database server. What we can get away with for now is the real question. I can offer up my boxes they are just running on Sata drives and currently running behind my 70/35 Verizon FiOS connection ( Jenkins.ovirt.info ). We could move them to a local co-lo for about $40.00 a U per month a friend of mine runs.
What does everyone else think? we can look at jenkins master load on jenkins.ovirt.org/monitoring (you need to be jenkins administrator to see it).
That is only monitoring the web traffic and not the actual build process and 3/4 of the data being pushed is archive.zip files for the node-iso builds. Although part of the build process it is the only part that shows up in the monitoring. And with nightly getting pushed to www.ovirt.org sometime this week the traffic should drop even lower. So far the only options I have seen talked about are. 1) limit what Jenkins can do by leaving it as is until hardware becomes available though *Red Hat in a few months. I don't think Jenkins will be able to make use of any more slaves since it is having a hard time even keeping up with the current number of slaves. 2) We build a 2nd master on EC2 and break the load up some. 3) Move to another VPS provider and see if Jenkins Master will run better on one of those services. Or if we can get the budget for it a dedicated box. 4) We find someone else who will donate hardware I would say 16G of Ram and either local storage or 10G storage network is pretty much min requirement 5) We use my hardware on ovirt.info and migrate the slaves over to it. Either using my current ISP that is fast but not on a static IP or Co-lo the boxes (About $80 a month). My boxes are dual quad core's with 16G of ram each but only have sata drives and no raid controller. *Since my understanding is Red Hat has promised hardware in the 2012 4th/2013 1st qtr we are looking to get by until then. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
Adding board list... I'm hoping with a wider audience, we might get additional feedback on hosting solutions. Mike On Tue, 2012-07-31 at 07:44 -0700, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
One idea is for us to throw out some favorite hosting providers here, and see if we can sort out what would be a good solution.
Other ideas have been floated.
Ideally, we'll get more hardware in the future via some hosting that Red Hat is working on providing for projects. In the meantime ...
Do any other sponsoring organizations have resources we can look in to?
Once we pick something, we can move it all fairly quickly, I think. Can we target the end of this week? Perhaps, if we don't run in to any complications ...
- Karsten _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
Itamar gave us this more direct list of what we need for hosting: We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves "For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk." "For jenkins master - quad core, 8-16GB RAM, a few disks for IO" "Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs." Are minimum need is: * A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently. A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.) - - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFQITiC2ZIOBq0ODEERAhzgAJ4+qEhOFhP7ULGkc6DDAroDEq9GKwCcDWRy jGSccfH9uvXh1WnnJ73vM7k= =qn5/ -----END PGP SIGNATURE-----
On 08/07/2012 06:47 PM, Karsten 'quaid' Wade wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
these are the slaves we currently have - we need many more to run CI on patches.
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO"
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves
I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.)
- - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iD8DBQFQITiC2ZIOBq0ODEERAhzgAJ4+qEhOFhP7ULGkc6DDAroDEq9GKwCcDWRy jGSccfH9uvXh1WnnJ73vM7k= =qn5/ -----END PGP SIGNATURE----- _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
On Tue, Aug 07, 2012 at 06:53:02PM +0300, Itamar Heim wrote:
On 08/07/2012 06:47 PM, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
these are the slaves we currently have - we need many more to run CI on patches.
Please remember that on more powerful hardware you can do more and EC2 is seriously underpowered compared to what you pay.
On 08/07/2012 07:29 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Aug 07, 2012 at 06:53:02PM +0300, Itamar Heim wrote:
On 08/07/2012 06:47 PM, Karsten 'quaid' Wade wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org.
Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
these are the slaves we currently have - we need many more to run CI on patches.
Please remember that on more powerful hardware you can do more and EC2 is seriously underpowered compared to what you pay.
trust me - i know. but i also know how this behaves with many more physical servers running build for each patch submitted, rebased, etc. running compile, unitests, findbugs, gwt compilation, etc.
On 08/07/2012 12:29 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Aug 07, 2012 at 06:53:02PM +0300, Itamar Heim wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
On 08/07/2012 06:47 PM, Karsten 'quaid' Wade wrote: these are the slaves we currently have - we need many more to run CI on patches. I don't believe we are going to get increased performance out of jenkins until we replace the master. Jenkins load avg is huge during normal operations just tiring to keep up with the slaves it has. There are offers out there to add slaves and that might be an option but the master just can't keep up. It is barely keeping up with it current load. Until the master is addressed there is only so much we can do with our current builds. Please remember that on more powerful hardware you can do more and EC2 is seriously underpowered compared to what you pay. Don't disagree. EC2 looks good on paper but the actual VM are way underpowered for what they claim you get.
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO" ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G of ram and non raided sata disks. The master is centos 6.3 and the slave is fedora 17. Between the two they are handling all the same build jobs as ovirt.org in about half the time. The reason why is 2 fold. 1) the signal sata drive is much faster then the EC2 storage so
On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote: the IO waits are limited on ovirt.info comparied to ovirt.org. 2 My master and slave are on the same gigabit network. So moving around files are 10x to 100x faster then ovirt.org.
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves
Ovirt.info tell me that a solid master with 16G of ram should be able to run 6 jobs and still feed several slaves so you could drop the 2 RHEL 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to replace the small Linode instance.
I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.)
I am open to this. It is my home so there is no backup generator or isp but my connection is 70/35M so it should be able to support jenkins needs. I have already given eedri root access to jenkins.ovirt.info incase that is a question in anyone mind. Thanks Robert
- - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iD8DBQFQITiC2ZIOBq0ODEERAhzgAJ4+qEhOFhP7ULGkc6DDAroDEq9GKwCcDWRy jGSccfH9uvXh1WnnJ73vM7k= =qn5/ -----END PGP SIGNATURE----- _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO" ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G of ram and non raided sata disks. The master is centos 6.3 and the slave is fedora 17. Between the two they are handling all the same build jobs as ovirt.org in about half the time. The reason why is 2 fold. 1) the signal sata drive is much faster then the EC2 storage so
On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote: the IO waits are limited on ovirt.info comparied to ovirt.org. 2 My master and slave are on the same gigabit network. So moving around files are 10x to 100x faster then ovirt.org.
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves
Ovirt.info tell me that a solid master with 16G of ram should be able to run 6 jobs and still feed several slaves so you could drop the 2 RHEL 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to replace the small Linode instance.
I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.)
I am open to this. It is my home so there is no backup generator or isp but my connection is 70/35M so it should be able to support jenkins needs. I have already given eedri root access to jenkins.ovirt.info incase that is a question in anyone mind.
a home based service for jenkins is a bit troubling for me. but maybe we can share some of the load - say, try to run on it some jobs at patch level and provide feedback on gerrit?
On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO" ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G of ram and non raided sata disks. The master is centos 6.3 and the slave is fedora 17. Between the two they are handling all the same build jobs as ovirt.org in about half the time. The reason why is 2 fold. 1) the signal sata drive is much faster then the EC2 storage so
On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote: the IO waits are limited on ovirt.info comparied to ovirt.org. 2 My master and slave are on the same gigabit network. So moving around files are 10x to 100x faster then ovirt.org.
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves
Ovirt.info tell me that a solid master with 16G of ram should be able to run 6 jobs and still feed several slaves so you could drop the 2 RHEL 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to replace the small Linode instance.
I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.)
I am open to this. It is my home so there is no backup generator or isp but my connection is 70/35M so it should be able to support jenkins needs. I have already given eedri root access to jenkins.ovirt.info incase that is a question in anyone mind.
a home based service for jenkins is a bit troubling for me. but maybe we can share some of the load - say, try to run on it some jobs at patch level and provide feedback on gerrit? That is doable. I also have a friend who can put the 2 boxes in a DC for 80 USD a month. If I could find raisers and a raid controller for
On 08/07/2012 12:25 PM, Itamar Heim wrote: the boxes that might also be an option. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, Aug 07, 2012 at 07:25:07PM +0300, Itamar Heim wrote:
On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO" ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G of ram and non raided sata disks. The master is centos 6.3 and the slave is fedora 17. Between the two they are handling all the same build jobs as ovirt.org in about half the time. The reason why is 2 fold. 1) the signal sata drive is much faster then the EC2 storage so
On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote: the IO waits are limited on ovirt.info comparied to ovirt.org. 2 My master and slave are on the same gigabit network. So moving around files are 10x to 100x faster then ovirt.org.
I'd rather have a master who runs no jobs itself. This also means it can be much lighter in terms of CPU and RAM. I expect the typical load on drives will also be easier since it's more sequential, but this is mostly speculation. It's also better from a security point of view since you run less untrusted code and it's much easier to reinstall a slave than a master. Obvious downside is more network traffic.
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves
Ovirt.info tell me that a solid master with 16G of ram should be able to run 6 jobs and still feed several slaves so you could drop the 2 RHEL 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to replace the small Linode instance.
I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.)
I am open to this. It is my home so there is no backup generator or isp but my connection is 70/35M so it should be able to support jenkins needs. I have already given eedri root access to jenkins.ovirt.info incase that is a question in anyone mind.
a home based service for jenkins is a bit troubling for me. but maybe we can share some of the load - say, try to run on it some jobs at patch level and provide feedback on gerrit?
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/07/2012 09:40 AM, Ewoud Kohl van Wijngaarden wrote:
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA.
My suggestion was just a stop-gap - run it at Robert's for a few weeks until we get new datacenter-based hosting. I like your idea of revamping the slave to be the new master. That gets us running faster soonest. Then we can use Robert's hosts as slaves, and also bring up a dedicated box at a hosting facility to run multiple slaves. - - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFQIUcH2ZIOBq0ODEERAsH/AKDJGYS1j7C/DTBpvF24ApvbNELVVgCghgv0 Suj9NSH6hrYiHPY3cuWb+Cw= =cNbu -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 08/07/2012 09:40 AM, Ewoud Kohl van Wijngaarden wrote:
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA. My suggestion was just a stop-gap - run it at Robert's for a few weeks until we get new datacenter-based hosting. Agreed I like your idea of revamping the slave to be the new master. That gets us running faster soonest. Then we can use Robert's hosts as slaves, and also bring up a dedicated box at a hosting facility to run multiple slaves. I don't. Jenkins master really needs to have the profile of a database server were the slaves don't need as much IO and can run fine on systems with limit IO. As a stop gap converting his slave to master might work but just like using the servers housed in my home is a stop gap I would
On 08/07/2012 12:49 PM, Karsten 'quaid' Wade wrote: put that in the same list. The limited IO we have seen from that VM would slow down everything just like the EC2 instance. Granted we don't need ssd raid 10 array in the box but a decent sata raid controller or even better a sas will allow faster builds and room for growth. Thanks Robert
- - Karsten - -- Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iD8DBQFQIUcH2ZIOBq0ODEERAsH/AKDJGYS1j7C/DTBpvF24ApvbNELVVgCghgv0 Suj9NSH6hrYiHPY3cuWb+Cw= =cNbu -----END PGP SIGNATURE----- _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, Aug 07, 2012 at 01:06:55PM -0400, Robert Middleswarth wrote:
On 08/07/2012 09:40 AM, Ewoud Kohl van Wijngaarden wrote:
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA. My suggestion was just a stop-gap - run it at Robert's for a few weeks until we get new datacenter-based hosting. Agreed I like your idea of revamping the slave to be the new master. That gets us running faster soonest. Then we can use Robert's hosts as slaves, and also bring up a dedicated box at a hosting facility to run multiple slaves. I don't. Jenkins master really needs to have the profile of a database server were the slaves don't need as much IO and can run fine on systems with limit IO. As a stop gap converting his slave to master might work but just like using the servers housed in my home is a stop gap I would put that in the same list. The limited IO we have seen from that VM would slow down everything just like
On 08/07/2012 12:49 PM, Karsten 'quaid' Wade wrote: the EC2 instance. Granted we don't need ssd raid 10 array in the box but a decent sata raid controller or even better a sas will allow faster builds and room for growth.
But does it also need much random IO when it's just master and not building? I think that uploading the artifacts is mostly sequential IO where SATA does a good enough job. I'm convinced that if we transform my slave into a master who's just a master we will have more than enough performance to last us for quite a bit.
On 08/07/2012 01:13 PM, Ewoud Kohl van Wijngaarden wrote:
On 08/07/2012 09:40 AM, Ewoud Kohl van Wijngaarden wrote:
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA. My suggestion was just a stop-gap - run it at Robert's for a few weeks until we get new datacenter-based hosting. Agreed I like your idea of revamping the slave to be the new master. That gets us running faster soonest. Then we can use Robert's hosts as slaves, and also bring up a dedicated box at a hosting facility to run multiple slaves. I don't. Jenkins master really needs to have the profile of a database server were the slaves don't need as much IO and can run fine on systems with limit IO. As a stop gap converting his slave to master might work but just like using the servers housed in my home is a stop gap I would put that in the same list. The limited IO we have seen from that VM would slow down everything just like
On 08/07/2012 12:49 PM, Karsten 'quaid' Wade wrote: the EC2 instance. Granted we don't need ssd raid 10 array in the box but a decent sata raid controller or even better a sas will allow faster builds and room for growth. But does it also need much random IO when it's just master and not building? I think that uploading the artifacts is mostly sequential IO where SATA does a good enough job. I'm convinced that if we transform my slave into a master who's just a master we will have more than enough
On Tue, Aug 07, 2012 at 01:06:55PM -0400, Robert Middleswarth wrote: performance to last us for quite a bit. What kind of storage does the box have? Remember the master is managing jobs so when several jobs finish at once they all are coping back there artifacts to the master. Although for the most part the jobs are sequential.
PS there are a few jobs it just makes since to run on master. Like the publishing one since it copies artifacts into a central folder and pushes them to www.ovirt.org. Thanks Robert
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
On Tue, Aug 07, 2012 at 01:24:40PM -0400, Robert Middleswarth wrote:
On 08/07/2012 09:40 AM, Ewoud Kohl van Wijngaarden wrote:
I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA. My suggestion was just a stop-gap - run it at Robert's for a few weeks until we get new datacenter-based hosting. Agreed I like your idea of revamping the slave to be the new master. That gets us running faster soonest. Then we can use Robert's hosts as slaves, and also bring up a dedicated box at a hosting facility to run multiple slaves. I don't. Jenkins master really needs to have the profile of a database server were the slaves don't need as much IO and can run fine on systems with limit IO. As a stop gap converting his slave to master might work but just like using the servers housed in my home is a stop gap I would put that in the same list. The limited IO we have seen from that VM would slow down everything just like
On 08/07/2012 12:49 PM, Karsten 'quaid' Wade wrote: the EC2 instance. Granted we don't need ssd raid 10 array in the box but a decent sata raid controller or even better a sas will allow faster builds and room for growth. But does it also need much random IO when it's just master and not building? I think that uploading the artifacts is mostly sequential IO where SATA does a good enough job. I'm convinced that if we transform my slave into a master who's just a master we will have more than enough
On Tue, Aug 07, 2012 at 01:06:55PM -0400, Robert Middleswarth wrote: performance to last us for quite a bit. What kind of storage does the box have? Remember the master is managing jobs so when several jobs finish at once they all are coping back there artifacts to the master. Although for the most
On 08/07/2012 01:13 PM, Ewoud Kohl van Wijngaarden wrote: part the jobs are sequential.
PS there are a few jobs it just makes since to run on master. Like the publishing one since it copies artifacts into a central folder and pushes them to www.ovirt.org.
Not sure of the exact configuration, but it's stored on one of our equallogic SATA SANs. It can't be worse than EC2, right ;)
On 08/07/2012 12:40 PM, Ewoud Kohl van Wijngaarden wrote:
On Tue, Aug 07, 2012 at 07:25:07PM +0300, Itamar Heim wrote:
On 08/07/2012 07:20 PM, Robert Middleswarth wrote:
On 07/31/2012 07:44 AM, Karsten 'quaid' Wade wrote:
We need to pick a new hosting solution for jenkins.ovirt.org. Itamar gave us this more direct list of what we need for hosting:
We currently have: - - Jenkins master - - Gerrit - - 2 fc17 jenkins slaves - - 2 rhel 6 jenkins slaves
"For gerrit I'd like a quad core, 8-16 GB RAM with decent IO from disk."
"For jenkins master - quad core, 8-16GB RAM, a few disks for IO" ovirt.info Jenkins is based on a pair of Dual quad core CPU's with 16G of ram and non raided sata disks. The master is centos 6.3 and the slave is fedora 17. Between the two they are handling all the same build jobs as ovirt.org in about half the time. The reason why is 2 fold. 1) the signal sata drive is much faster then the EC2 storage so
On 08/07/2012 11:47 AM, Karsten 'quaid' Wade wrote: the IO waits are limited on ovirt.info comparied to ovirt.org. 2 My master and slave are on the same gigabit network. So moving around files are 10x to 100x faster then ovirt.org. I'd rather have a master who runs no jobs itself. This also means it can be much lighter in terms of CPU and RAM. I expect the typical load on drives will also be easier since it's more sequential, but this is mostly speculation.
It's also better from a security point of view since you run less untrusted code and it's much easier to reinstall a slave than a master. Obvious downside is more network traffic.
"Jenkins slaves - depends on pricing i guess, but we need quite a few of these to run the jobs."
Are minimum need is:
* A new bare metal host for Jenkins master. * A new bare metal or VM host for Gerrit. * 4 VMs for Jenkins slaves Ovirt.info tell me that a solid master with 16G of ram should be able to run 6 jobs and still feed several slaves so you could drop the 2 RHEL 6.x slaves and just need 2 slaves for Fedora 17 builds plus a VM to replace the small Linode instance. I'm going to see what we can do with ~$150/mon, with the idea that we could replace the relatively small Linode instance we're spending ~$40/mon on currently.
A crazy idea - should we move the master to jenkins.ovirt.info right away to get the performance boost, as a stop-gap until we get a dedicated host in a datacenter? (Just trying to make us not dependent on Robert's house as if it's a datacenter.) I am open to this. It is my home so there is no backup generator or isp but my connection is 70/35M so it should be able to support jenkins needs. I have already given eedri root access to jenkins.ovirt.info incase that is a question in anyone mind. a home based service for jenkins is a bit troubling for me. but maybe we can share some of the load - say, try to run on it some jobs at patch level and provide feedback on gerrit? I agree with your concerns on the home based service for something this critical as a master. One solution could be to revamp the slave I provide into a master and use Roberts infra as slaves. It's a VM running on our production RHEV cluster and has sufficient HA. My concern with that is the current slave seems to have slow IO. I don't really want to replace 1 slow IO monster with another slow IO monster. Trust me I understand the desire to not run the master on something not in a proper DC. I think Itamar Heim idea of dividing the load between the boxes makes since. We can keep the important builds on ovirt.org and move the per patch builds to ovirt.info until we can build a proper Jenkins cluster.
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Thanks Robert Middleswarth @rmiddle (twitter/IRC)
participants (6)
-
Ewoud Kohl van Wijngaarden -
Eyal Edri -
Itamar Heim -
Karsten 'quaid' Wade -
Mike Burns -
Robert Middleswarth