[lug] [Slightly OT] File Management?

[Lee Woodworth]

Rob Nagler wrote:
> Lee Woodworth writes:
>> It sounds like you have looked at rsync and decided it isn't useful
> 
> We use rsync to mirror the data.

You must be doing humongous backups with a large percentage change of
of data between backups.

Is it a hard requirement to keep point-in-time views of the whole
backup set for each day? If not, then maybe this is something to
consider (this is essentially what my backup process is):

for d in /root /etc /data ....; do
     /bin/mkdir -p /backup/history/server1/20090324/$d
     /bin/chmod 0700 /backup/history/server1/20090324/$d
     /usr/bin/rsync -v --archive --stats --delete --backup
          --backup-dir=/backup/history/ruby/20090324/$d $d/ /backup/server1/$d/

     /usr/bin/find $d -l >> /backup/history/ruby/20090324/files.lst

     #Remove old history dirs
     /bin/rm -rf --preserve-root /backup/history/server1/20090220/$d
done

This keeps the complete backup copy in sync with master and keeps a per-day
directory tree of changed files between backups. Allows for handling "oops,
deleted file ...." and does have list of the files at the time of the backup.

For my small backups (56GB) this runs in 3min to a usb disk (but only 0.5% changed
files) - its ~225k files with a variety of file sizes (with lots of small files)