[lug] Modern day benchmarking and sizing

[Maxwell Spangler]

On Tue, 2010-07-13 at 11:17 -0600, Stephen Queen wrote:

> My question was general. I will have more than one application in the
> future. The question is how does a person go about determining what
> cpu/main board is best for their app. Not is this particular cpu
> appropriate for this particular app. It is probably to general of a
> question to ask on a mailing list, and expect a reasonable answer.

This is a general question, but a very important question in the era of
virtualization that we're entering into.

In the past, when users were dissatisfied with performance, we threw
faster hardware at the problem.  But we reached ceilings in how fast we
could get cpu, memory and disk systems to perform and we ended up with
lots of power-hungry, heat-producing servers taking up a lot of space.

The trend being promoted for the future is to have those
high-performance systems available but used only when needed so that
applications can be consolidated to few servers in non-peak periods and
expanded to many servers during demanding periods.  This lets machines
be powered off to save energy, reduced cooling needs and replaces bulky
single and multi-processor systems with multi-core systems that may take
up less physical space.

But in order to work with systems dynamically, you need to understand
more than ever how an application works and how to monitor it for
satisfactory performance.  If you can give it only what it needs yet
still achieve satisfying performance, you can consolidate more servers
onto fewer machines.  But if you do that too aggressively, you choke the
applications of resources and fail to satisfy your users.

So in this way I see attempting to choose between Atom processors and
desktop/server processors as similar to choosing between virtual
machines and dedicated machines.  It's easy just to throw as much power
as you can afford at the problem, but there are tremendous benefits to
being efficient when you do it right.

I approach it like this:

1. Understand the basic technology: is this app a file server, a web
server, a database server?  A general idea of what the app is sets your
perspective for further investigations.

2. What are the technical details of it?  Is it an interpreted Python
program or a compiled C program?  Does it use a lot of disk activity?
Does it manipulate huge chunks of memory?  These direct the path to
success such as: If you need to manipulate 6GB graphics images, skip
32bit processors and go straight to 64bit processors that can handle
that amount of memory easily.

3. What does the community know about it?  One of the biggest changes in
computing in the last ten years is the move to collaborate openly, by
default.  So find out what others already know such as guidelines for
sizing, what attempts fail so you can skip trying them, etc.  But don't
trust these as absolutes, just things to consider.

4. Finally, do your own benchmarking and testing.  The only benchmark
results that will ever really matter are those of the problem you're
trying to solve.  So apply the ideas from steps 1-3, benchmark different
configurations, and share with the community to revise your tests.
Repeat until you reach satisfactory results, if possible.

I'm eager to hear what others have to say on this.

-- 
Maxwell Spangler
========================================================================
        Linux, Unix and Database Administration
        Currently: Boulder, Colorado
        LinkedIn: http://www.linkedin.com/in/maxwellspangler