[lug] Limiting GoogleBot's Hoovering

[Bill Thoen]

Is there any way to stop Googlebots from hoovering my public FTP site
multiple times in a session? Last night, one of Google's robots downloaded
everything on my site 6 times. I don't mind PEOPLE downlaoding what they
WANT, but it sort of bugs me when a machine just takes everything and then
deos it several times at a go. I'd like to know what's popular and what's
not because it helps me figure out what to post, but this sort of mindless
downloading is not what I want.

- Bill Thoen