[lug] Robust storage

[Sean Reifschneider]

On Mon, May 02, 2005 at 12:25:09AM -0600, Dean Brissinger wrote:
>In general I avoid software raid for valuable data storage.  Hardware
>raid solutions are built for situations like a power failure.  They

Yeah, livejournal had such good luck with their systems during their recent
power-failure...

To be honest, I've actually been moving away from hardware RAID these days.
Even with 3ware, where the Linux driver support is pretty good, we've just
had problems where the utilities aren't up to it and lead to problems
requiring booting into the BIOS or the like.

I've had extremely good luck with Linux software RAID over the same time
period.  Oh, add to it that the software RAID is well documented, if in no
other way than through the code.  A local guy lost his 3ware RAID array in
his home box last summer, it just got confused and he couldn't recover it.
He basically had the data, but needed to tell the controller, through
poking appropriate bits onto a new drive, to bring the array back up, but
3ware wouldn't cooperate with documenting those parts of the drive.

>solve data loss by using an on-board battery to save the write cache
>until power returns.  Even with a UPS the hardware cache is better
>because it will ensure write-through integrity if a disk fails.

Really, the battery backed up cache on the controller is for performance,
not safety.  It allows the controller to confirm data written to the disc
when it gets into the cache, instead of the normal case where it waits for
it to get on the disc.  The problem here is the added complexity may lead
to other subtle problems, as we saw with livejournal.

>original data was bad.  ext2/3 are likely culprits of minor filesystem
>corruption.  Mostly I have found problems with ext* when restoring up

We've had this one particular system for a client that had horrible
problems with ext3.  I'm thinking it was related to the pattern of usage,
but I've never been able to reproduce it.  Perhaps it's been fixed, or
perhaps there was something else at work.  We were seeing filesystem
corruption pretty much every week, as found by the data validation tool.
This was without unclean shutdowns, and fsck being run every time.
Switching to JFS solved the problem, there has maybe been once instance of
corruption in the 3 years since.

>very large filesystems.  My current favorate solution is SuSE Enterprise
>Server 9 with reiserfs.  I have tested it by verifying the filesystem

I need to try reiser again.  I steered clear of it for quite a while after
a year where all of the tummy.com folks ran into serious file-system
corruption on our laptops running Reiser.  That was around 4 years ago now
though.  It's time to try again.

>Search google for open source backup solutions.  You are looking for
>good file integrity verification features.  Even bit-wise if you so
>desire.  The value of the data should determine the amount of hoops you

The real trick is to run regular validations against the backups.  Files
which have changed but have the same mtime probably indicate some problems.

Sean
-- 
 If Rear Admiral Grace Murray Hopper spoke out against the MPAA:
 "A movie on DVD is safe, but that's not what movies are for."
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995.  Qmail, Python, SysAdmin