[lug] Robust storage

[Sebastian Sobolewski]

D. Stimits wrote:

> Daniel Webb wrote:
>
>> I just noticed I have file corruption in an old mailing list archive 
>> (gzip
>> fails about 1/4 of the way through).  It's not one I care about much,
>> but it's got me thinking about the issue in general.  I have no idea
>> when this corruption happened, sometime in the last two years.  Here are
>> some questions I have for the experts out there:
>>
>>  * How can I know if files have been corrupted through hardware errors?
>>    Would Linux software RAID have prevented this?
>
>
> I doubt RAID would help. If anything it would add complications, since 
> your corruption was at the file level and not the filesystem. RAID can 
> build copies and thus detect mirror errors, which would possibly aid 
> if say a disk defect caused the error...but most defects don't simply 
> show up as a bit alteration, they tend to show up as fatal filesystem 
> errors (which might be fixed by bad block relocations).

Actually it's worse then that.  I've had a case were reading from one 
mirror returned different data then from the other in a raid 1 set.  ( 
One of the disks was returning a silent,uncorrected bit error on 3 bits 
in a single block ) RAID1/RAID5 are great for surviving single disk 
crashes, but you can't depend on it for anything else.  For instance if 
the FS goes bad, you now have 2 copies of the badness.

>
>>  * How can I know if files have been corrupted by bugs in the low-level
>>    block drivers (the filesystem drivers or in my case drbd)?
>>    Would Linux software RAID have prevented this?  What happens if the
>>    corruption is cause by the RAID driver?
>
>
> I'm guessing you need more of a filesystem journaling and that RAID 
> itself will only help with overt failures. You can use ECC ram to 
> avoid random bit failures in RAM (and believe it or not, apparently 
> radiation hitting the ground from space, cosmic rays, can cause a 
> random rare bit alteration and does so more frequently at higher 
> altitudes like the Boulder area versus sea level).
>
>>  * What are some inexpensive solutions to this problem?
>
>
> Journaling filesystems, and UPS that guarantee full power during 
> brownouts and low voltage situations. Brownouts and low voltages in 
> general are a big danger to data corruption because the hardware 
> doesn't necessarily know voltage is too low, and data can be altered 
> without actual failure. Not all UPS provide power during low voltage, 
> cheaper UPS will provide power only during overt failure. Never go 
> without an UPS and never use an UPS that doesn't handle brownouts. On 
> a similar line, if you have a power supply that is marginal and can 
> under some circumstances act like a self-contained undervoltage (such 
> as surge current use by a CD drive or HD starting up), an UPS won't 
> help. Don't use an underrated power supply.
>
> Among journaling filesystems, beware that most all journal metadata 
> only, and not full data (full data journaling is a huge performance 
> hit), so it can restore to a particular time but not necessarily all 
> data will be available that was writing at the moment of failure...you 
> might lose a few seconds, but the filesystem will be undamaged. But 
> then if you try to write garbage you'll just get a good copy of 
> garbage anyway.

Journaling helps make sure you don't hose an entire Filesystem when the 
machine takes a power hit during a lot of writing. But you can still and 
likely WILL corrupt any files that are in mid-write.  Current generation 
disk drives have 16MB worth of cache on board, anything in that cache 
will be gone on a power hit.. that's a lot of data. ( Of course you 
could always turn the caches off and take the performance hit, highly 
recommended if you care about your files )

Here is my paranoid setup:

2 systems, each with a disk arrays running raid5.  The systems have 
duplicates of all files I consider critical rsynced to each other. Both 
systems are running with disk caches disabled, ext3 with journal and 
attached to a UPS ( configured to shutdown if there is less then 15 min 
of battery left ) .  All files have MD5 hashes calculated and the hashes 
are validated to make sure any files that have not changed in the last 
day match their hash. ( IE the file did not get silently corrupted 
overnight ) And weekly backups to DVD of new files. I also run badblocks 
on the disk drives every few weeks.

But this may be overkill for most situations.


>
> D. Stimits, stimits AT comcast DOT net
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#colug
>