[lug] [OT] Web site dead link checking

[Daniel Webb]

On Thu, Nov 16, 2006 at 12:29:16AM -0700, Daniel Webb wrote:

> There are seemingly endless numbers of programs and web services for web site
> maintenance like dead link checking.  What are the best ones?  What other
> features should I consider while I'm at it (maybe validation?)  I prefer easy
> but more importantly I want it to just work once set up and keep working with
> no hassle.  I prefer a local Free software package, but would consider an
> online free as in beer service.

(responding to myself)

I checked out all the Free ones, and I like linkchecker for simple use.  I wrote
this short bash script to run it once a day and email me the report only if
there are more than N errors [BCC'ed to the linkchecker author]:

# ----------------------------------------------------------
# Check web site links once per day
#   Return 0
# arguments:
#   $1 - web site URL
#   $2 - notification email
#   $3 - threshold number of errors
#
function die() { echo "$0: $*"; exit 1; }
function check_web_links()
{
logfile=/tmp/linkchecker.log
[ -z "$1" -o -z "$2" -o -z "$3" ] && die "check_web_links requires two arguments"
do_check=false
if [ ! -f $logfile ]; then
    do_check=true
else
    # Has it been at least a day since last check?
    find $logfile -mtime +1 | grep link && do_check=true
fi
if [ $do_check = true ]; then
    linkchecker $1 >$logfile 2>/dev/null
    errors=$(grep Error: $logfile | wc -l)
    if [ $errors -gt $3 ]; then
        cat $logfile | mail -s "linkchecker: more than $3 errors" $2
    fi
fi
return 0
}