virtualization benchmark suites
Hi folks, I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt. IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using transaction and networking workloads. I'm not proposing DayTrader. But IBM does have some distributed benchmarks that may be interesting to the community. And others of you probably also have similar test suites. What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea. Thanks, Mike Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday@us.ibm.com http://code.ncultra.org
Personally I think that would be great. It would be good to also have benchmarks that can cover the full stack, so people can get an understanding of scaling the full management platform + KVM for the data centre. Carl. On 09/22/2011 05:01 PM, Michael D Day wrote:
Hi folks,
I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt.
IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using transaction and networking workloads.
I'm not proposing DayTrader. But IBM does have some distributed benchmarks that may be interesting to the community. And others of you probably also have similar test suites.
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
Thanks,
Mike
Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday@us.ibm.com http://code.ncultra.org
_______________________________________________ Project-planning mailing list Project-planning@ovirt.org http://lists.ovirt.org/mailman/listinfo/project-planning
-----Original Message----- From: project-planning-bounces@ovirt.org [mailto:project-planning-bounces@ovirt.org] On Behalf Of Michael D Day Sent: Friday, September 23, 2011 0:02 AM To: project-planning@ovirt.org Subject: virtualization benchmark suites
Hi folks,
I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt.
IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using
transaction and networking workloads.
I'm not proposing DayTrader. But IBM does have some distributed
benchmarks that may be interesting to the
community. And others of you probably also have similar test suites.
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
How would that differentiate from specvirt?
Thanks,
Mike
Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday@us.ibm.com http://code.ncultra.org
On 09/22/2011 04:11 PM, Itamar Heim wrote:
-----Original Message----- From: project-planning-bounces@ovirt.org [mailto:project-planning-bounces@ovirt.org] On Behalf Of Michael D Day Sent: Friday, September 23, 2011 0:02 AM To: project-planning@ovirt.org Subject: virtualization benchmark suites
Hi folks,
I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt.
IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using transaction and networking workloads.
I'm not proposing DayTrader. But IBM does have some distributed benchmarks that may be interesting to the community. And others of you probably also have similar test suites.
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
How would that differentiate from specvirt?
specvirt is neither free as in beer nor free as in speech. It's not something that a customer can easily obtain and play around with themselves. Regards, Anthony Liguori
Thanks,
Mike
Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday@us.ibm.com http://code.ncultra.org
_______________________________________________ Project-planning mailing list Project-planning@ovirt.org http://lists.ovirt.org/mailman/listinfo/project-planning
On 09/23/2011 12:20 AM, Anthony Liguori wrote:
On 09/22/2011 04:11 PM, Itamar Heim wrote:
-----Original Message----- From: project-planning-bounces@ovirt.org [mailto:project-planning-bounces@ovirt.org] On Behalf Of Michael D Day Sent: Friday, September 23, 2011 0:02 AM To: project-planning@ovirt.org Subject: virtualization benchmark suites
Hi folks,
I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt.
IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using transaction and networking workloads.
I'm not proposing DayTrader. But IBM does have some distributed benchmarks that may be interesting to the community. And others of you probably also have similar test suites.
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
How would that differentiate from specvirt?
specvirt is neither free as in beer nor free as in speech. It's not something that a customer can easily obtain and play around with themselves.
Does such benchmark ever existed (apart from kernel build)? :) As you pointed Anthony, let the ovirt project first focus on simplify the build process before we develop a competing benchmark suite to the one that we base lots of our marketing over. Sorry spilling cold water, just one thing at a time.
Regards,
Anthony Liguori
project-planning-bounces@ovirt.org wrote on 09/22/2011 05:34:38 PM:
Does such benchmark ever existed (apart from kernel build)? :) As you pointed Anthony, let the ovirt project first focus on simplify the build process before we develop a competing benchmark suite to the one that we base lots of our marketing over.
Yes, there are several distributed benchmarks that test different aspects of virtualization performance. SPECVirt tests server consolidation. It doesn't test memory over-commitment. In fact, I don't think anyone who publishes a SPECVirt score using a configuration that over-commits memory because it hurts the score. There are other benchmark suites that test HPC cluster-style workloads running on a hypervisor. And networking performance. These are just some of the ones we use within IBM. They are very useful. If we would have had better benchmark suites to test memory usage you would have decided to include the balloon driver in RHEL 5.4 instead of relying solely upon KSM. And if we had used a good distributed transactional benchmark like DayTrader we would have known early on that small-packet network performance is relatively poor. Just a couple of examples. If a benchmarking project does happen, it would need to be a self-sustaining project under the oVirt community umbrella. That means there would need to be interest in the form of independent developer resources. And keep in mind I'm passing on customer feedback. (I happen to think its a good suggestion as well). Thanks, Mike Mike Day IBM Distinguished Engineer Chief Virtualization Architect, Open Systems Development Cell: +1 919 371-8786 | mdday@us.ibm.com http://code.ncultra.org
On 09/22/2011 05:34 PM, Dor Laor wrote:
On 09/23/2011 12:20 AM, Anthony Liguori wrote:
On 09/22/2011 04:11 PM, Itamar Heim wrote:
-----Original Message----- From: project-planning-bounces@ovirt.org [mailto:project-planning-bounces@ovirt.org] On Behalf Of Michael D Day Sent: Friday, September 23, 2011 0:02 AM To: project-planning@ovirt.org Subject: virtualization benchmark suites
Hi folks,
I spoke with several financial customers interested in KVM earlier this week and two topics we discussed were KVM benchmarking and the oVirt community. Two of the customers suggested that we contribute benchmarking suites to oVirt as projects, and that we form a benchmarking community under oVirt.
IBM has successfully contributed a benchmark called DayTrader to the Apache foundation and formed a community around that benchmark, which tests middleware using transaction and networking workloads.
I'm not proposing DayTrader. But IBM does have some distributed benchmarks that may be interesting to the community. And others of you probably also have similar test suites.
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
How would that differentiate from specvirt?
specvirt is neither free as in beer nor free as in speech. It's not something that a customer can easily obtain and play around with themselves.
Does such benchmark ever existed (apart from kernel build)? :) As you pointed Anthony, let the ovirt project first focus on simplify the build process before we develop a competing benchmark suite to the one that we base lots of our marketing over.
Sorry spilling cold water, just one thing at a time.
My view is build the eco-system. That means welcome any additional complementary project. Having multiple projects under coordination means that each project can build at it's own rate, yet we can have integration. Additionally, not everyone can work on the same thing. We have the concept of project maturity, so there is no risk as long as it is well defined and complementary. Additionally, if a project comes in and after a year or so gets no traction we can sunset it. This provides a healthy model to have community innovation in and around the oVirt core project. regards Carl.
project-planning-bounces@ovirt.org wrote on 09/23/2011 07:57:08 AM:
From: Carl Trieloff <cctrieloff@redhat.com>
....
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea.
How would that differentiate from specvirt?
specvirt is neither free as in beer nor free as in speech. It's not something that a customer can easily obtain and play around with themselves.
Does such benchmark ever existed (apart from kernel build)? :) As you pointed Anthony, let the ovirt project first focus on simplify the build process before we develop a competing benchmark suite to the one that we base lots of our marketing over.
Sorry spilling cold water, just one thing at a time.
+1 (to the extent I get a vote.. Hmm, may need to go look at those rules again! :-) )
My view is build the eco-system. That means welcome any additional complementary project. Having multiple projects under coordination means that each project can build at it's own rate, yet we can have integration. Additionally, not everyone can work on the same thing. We have the concept of project maturity, so there is no risk as long as it is well defined and complementary.
Additionally, if a project comes in and after a year or so gets no traction we can sunset it. This provides a healthy model to have community innovation in and around the oVirt core project.
We need to get a lot of seeds going.. while additional projects may take a little attention from the initial project base, we need to foster growth. I think the real question is whether or not we think it's within the scope and mission of oVirt; I think there's certainly a need for it as a community effort.. certainly important to the KVM ecosystem in general.. may not exactly fall into 'management' related, but I think we're broader than that. It's also somewhat independent of the core framework, so hopefully doesn't cause too much disruption to getting the base going, and probably doesn't pull on/distract the core buildout.. may appeal more to somewhat of a different crowd..
regards Carl.
Cheers, Frank
On 09/22/2011 09:54 PM, Frank Novak wrote:
project-planning-bounces@ovirt.org wrote on 09/23/2011 07:57:08 AM:
From: Carl Trieloff <cctrieloff@redhat.com> ....
What is the group's feeling about benchmarking projects? If there's interest I'll pursue things on our end (no promises). But I think this is good feedback from our customers and am interested to hear comments on the idea. How would that differentiate from specvirt? specvirt is neither free as in beer nor free as in speech. It's not something that a customer can easily obtain and play around with themselves. Does such benchmark ever existed (apart from kernel build)? :) As you pointed Anthony, let the ovirt project first focus on simplify the build process before we develop a competing benchmark suite to the one that we base lots of our marketing over.
Sorry spilling cold water, just one thing at a time.
+1 (to the extent I get a vote.. Hmm, may need to go look at those rules again! :-) )
My view is build the eco-system. That means welcome any additional complementary project. Having multiple projects under coordination means that each project can build at it's own rate, yet we can have integration. Additionally, not everyone can work on the same thing. We have the concept of project maturity, so there is no risk as long as it is well defined and complementary.
Additionally, if a project comes in and after a year or so gets no traction we can sunset it. This provides a healthy model to have community innovation in and around the oVirt core project.
We need to get a lot of seeds going.. while additional projects may take a little attention from the initial project base, we need to foster growth. I think the real question is whether or not we think it's within the scope and mission of oVirt; I think there's certainly a need for it as a community effort.. certainly important to the KVM ecosystem in general.. may not exactly fall into 'management' related, but I think we're broader than that. It's also somewhat independent of the core framework, so hopefully doesn't cause too much disruption to getting the base going, and probably doesn't pull on/distract the core buildout.. may appeal more to somewhat of a different crowd..
yes, the key is going to be working out what is complementary, balancing building momentum and breadth. I'm sure we will spend many emails on this topic working through this as the project grows. I would also like some of Jim's input here when he gets back from PTO. Carl.
participants (6)
-
Anthony Liguori -
Carl Trieloff -
Dor Laor -
Frank Novak -
Itamar Heim -
Michael D Day