[lug] compressed cpio archive problems

[Sean Reifschneider]

On Sun, Nov 18, 2001 at 09:07:28AM -0700, S. Luke Jones wrote:
>and the man page is pretty unhelpful. (It says zcat will read up
>to the point of failure, but no hints as to how to skip the bad
>part and continue on after. Perhaps that's not possible with .Z

Compressed data after a missing or corrupt block can be thought of as
effectively encrypted.  You may be able to attack it that way.  The problem
being that the tables that drive the compression in lzw are based on the
preceeding data, so if you miss some of that data the table gets out of
sync.

There are block-based compression schemes that effectively dump their
tables periodically, so that the reader can re-sync and continue reading
compressed data after the bad part.  "compress" didn't use such a
configuration, however.

For reliability, the best way to go is to compress the files, then put them
in the archive and leave the archive uncompressed.  Yes, you sacrafice some
compression this way, but it's very easy to recover.

I've been meaning to start working on my version of "tar" again, which
would have an extension to do exactly this.  A new file type that would be
"data compressed by the archiver", so it would be as easy to use as
tar/gzip, but would be more robust and would still have the index readable
without compressing.  This is for a backup system I'm dealing with where it
would be nice to have the clients compress the data, but the server
processes the stream as it comes through, and builds an index for fast
recovery and the like.

Sean
-- 
 I love the smell of napalm in the morning.  It smells like...  Victory.
               -- _Apocalypse_Now_
Sean Reifschneider, Inimitably Superfluous <jafo at tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python