[lug] Google (and other) ftp crawlers.

[Will]

http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

On Fri, Feb 25, 2011 at 1:46 PM, Dave Pitts <dpitts at cozx.com> wrote:

> Hello:
>
> Is there a way to get Google (and other) sites to stop crawling through my
> anon
> ftp site? They download everything and drop my network access to a
> crawl....
> sometimes causing my applications to timeout and die.
>
> Thanks in advance.
>
> --
> Dave Pitts             PULLMAN: Travel and sleep in safety and comfort.
> dpitts at cozx.com        My other RV IS a Pullman (Colorado Pine).
> http://www.cozx.com
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20110225/b12605ed/attachment.html>