[lug] Software RAID5 performance

[Jeff Schroeder]

Hey all--

I just set up a fileserver (Samba+Appletalk+NFS) and configured a trio 
of 120GB drives in a software RAID5 configuration.  This gives me 240GB 
of storage, and I have an 80GB boot drive (rather than booting from 
RAID).

I was curious how the software RAID would perform, so I did some simple 
testing.  First I created a 10MB file:

# dd if=/dev/zero of=/tmp/bigfile bs=1024 count=10000

Note that /tmp is tmpfs, meaning it's actually a memory-based filesystem 
(and thus very fast).  The other two drives were both formatted as 
ext3.  Now I copied the file to three places: the /tmp area (in 
memory), the boot drive (no RAID), and the data drive (RAID5).  The 
command I used in each case was:

# time cp /tmp/bigfile /xxx (where xxx is the appropriate drive)

The results for tmpfs:
real    0m6.937s
user    0m0.002s
sys     0m0.084s

The results for the boot drive:
real    0m5.918s
user    0m0.001s
sys     0m0.085s

And the results for the RAID drive:
real    0m4.537s
user    0m0.001s
sys     0m0.084s

I did the copy test several times to make sure it was consistent, and 
these numbers are representative of the performance I saw.  Now, I'm a 
little baffled here because they're exactly *opposite* what I expected!  
I would think the RAM-based tmpfs would be the fastest, and the RAID 
drive would be the slowest (since the kernel has to calculate parity 
information on every block write).

It occurred to me that perhaps the file I created (10MB of zeros) wasn't 
"realistic", so I grabbed another file off the system that was around 
10MB in size.  The results were the same, with RAID outperforming the 
others.

Does anyone have any ideas why things look like this?  Am I 
misunderstanding the way tmpfs works, or does the kernel somehow treat 
RAID drives in a special way to improve performance?

One other related question: since setting up the RAID5, I've noticed the 
system load on this machine is fairly high-- typically between 1 and 2 
when it's being lightly used (meaning someone is copying files via 
Samba), and often over 5 during heavier use.  I can only assume this is 
because the CPU is doing parity calculations.

Does anyone have similar experience?  Can you confirm that the CPU load 
is typically high, or is it possible I've mis-configured something 
along the line?  (I've been using software RAID1 for years without a 
hiccup; this is my first foray into RAID5.)

TIA,
Jeff