[lug] ftp problem

[George Sexton]

Here's a script I use to push tar files over ssh for remote backup. It does
a complete backup once a week, and a differential the rest of the time. The
one kind of plus is that the tar file doesn't have to reside on the local
machine for transmit.


#############
Rbackup.sh
#############
#!/bin/sh

. /root/bin/rbackup.cfg

TAR_OPTS="--preserve --exclude-from $EXCLUDELST -cT $PATHLST"
find ~ -iname ".LAST_FULL" -mtime +6 -exec rm -f "{}" \;
if [ -f $MARKER_FILE ]; then
	STYPE="differential"
	TAR_OPTS="--newer=$MARKER_FILE $TAR_OPTS"
else
	touch $MARKER_FILE
	STYPE="full"
fi
OUTFILE="$REMOTEPATH/`hostname -s`-$STYPE-`date +%Y%m%d`.tar.bz2"
if [ ! -z $REMOTE_PORT ]; then
	SSH_OPT="-p $REMOTE_PORT"
fi
tar $TAR_OPTS | bzip2 | ssh -Te none $SSH_OPT $REMOTEUSER@$REMOTEHOST "dd
bs=1024 of=$OUTFILE"

############
#Rbackup.cfg
############
# The format of paths.lst is
#	Local Path
#	Use Compression (Y or N)
#

PATHLST=/root/bin/paths.lst
EXCLUDELST=/root/bin/exclude.lst

REMOTEHOST=xxx.xxxx.com
REMOTEPATH=/media/bkup/remote
REMOTEUSER=yyyyyyyyyy

COMPRESSION_PROGRAM=bzip2

#REMOTE_PORT=2022
MARKER_FILE=/root/.LAST_FULL





George Sexton
MH Software, Inc.
http://www.mhsoftware.com/
Voice: 303 438 9585
  

> -----Original Message-----
> From: lug-bounces at lug.boulder.co.us 
> [mailto:lug-bounces at lug.boulder.co.us] On Behalf Of Gary Hodges
> Sent: Wednesday, March 09, 2005 8:36 AM
> To: Boulder (Colorado) Linux Users Group -- General Mailing List
> Subject: Re: [lug] ftp problem
> 
> I could try it, but I think I have conclusively shown that 
> the problem 
> is in the transmission, not inherent in the files.  When a 
> file has come 
> in with errors I've verified it is OK on the machine it was 
> created on.  
> And often with a second attempt it comes in OK.
> 
> Cheers,
> Gary
> 
> George Sexton wrote:
> 
> >What if you try:
> >
> >tar -c path | bzip2 > file.tar.bz2
> >
> >And then transmit it.
> >
> >George Sexton
> >MH Software, Inc.
> >http://www.mhsoftware.com/
> >Voice: 303 438 9585
> >  
> >
> >  
> >
> >>-----Original Message-----
> >>From: lug-bounces at lug.boulder.co.us 
> >>[mailto:lug-bounces at lug.boulder.co.us] On Behalf Of Gary Hodges
> >>Sent: Tuesday, March 08, 2005 4:32 PM
> >>To: Boulder (Colorado) Linux Users Group -- General Mailing List
> >>Subject: Re: [lug] ftp problem
> >>
> >>For another test I tried uploading ten uncompressed tar files 
> >>and four 
> >>came in without problems.  Tar archives are sequential, 
> >>right?  I think 
> >>it is strange that files are randomly missing from the tar 
> >>archives.  If 
> >>the transfer didn't finish, wouldn't the files be missing 
> >>from the end?  
> >>I'll play with the size verification code below and the md5 
> >>suggestion 
> >>too...
> >>
> >>Gary
> >>
> >>Ron Wright wrote:
> >>
> >>    
> >>
> >>>Also, at ftp time you can verify the size on both ends.  Use:
> >>>  $localsize = (stat($filename))[7];
> >>>  $remotesize = $ftp->size($filename);
> >>>
> >>>
> >>>On Tue, 8 Mar 2005 12:05:06 -0700, Daniel Webb 
> >>>      
> >>>
> >><lists at danielwebb.us> wrote:
> >>    
> >>
> >>> 
> >>>
> >>>      
> >>>
> >>>>On Tue, Mar 08, 2005 at 10:36:11AM -0700, Gary Hodges wrote:
> >>>>
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>>>Is it possible the firewall is stripping files out of the 
> >>>>>          
> >>>>>
> >>tar archive or
> >>    
> >>
> >>>>>should we look somewhere else?
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>>My guess is that the missing files are at the end of the 
> >>>>        
> >>>>
> >>archive, and
> >>    
> >>
> >>>>the archives are sometimes not getting through completely.  
> >>>>        
> >>>>
> >>Maybe add
> >>    
> >>
> >>>>something to your script to send along an md5 hash to 
> make sure the
> >>>>received file is the same as the sent file.  The last 
> place I would
> >>>>suspect would be that a firewall is set up to examine and 
> >>>>        
> >>>>
> >>filter .tar.gz
> >>    
> >>
> >>>>files...
> >>>> 
> >>>>
> 
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#colug
> 
>