Stop Web Crawlers?

Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

Ktsmom02

Thread Starter
Joined
Jul 5, 2002
Messages
38
My bandwidth is being used up by bots & web crawlers! How do I stop this? This is the second month I've been notified that I'm over 80% usage. Last month, it locked up my site for 4 days. I need a really simple explanation on how to block these nuts! :(
 

tomdkat

Retired Trusted Advisor
Joined
May 6, 2006
Messages
7,148
Creating a robots.txt file is another option to consider. This can be used to easily control web crawlers and robots.

Peace...
 

tomdkat

Retired Trusted Advisor
Joined
May 6, 2006
Messages
7,148
Some malicious bots will ignore robots.txt, so by denying them access in .htaccess is more effective ;)
True but using a robots.txt file is far easier than creating rules to block bots. If the majority of the bots in question are "rogue", than a robots.txt file won't be of much help. If the majority of the bots in question aren't rogue, then a robots.txt file can be used to easily control which bots can crawl the site and what areas of the site they are permitted to crawl. Doing this doesn't sacrifice all benefits of allowing a bot to crawl the site if some search engine presence is desirable.

Using a combination of both might be beneficial since malicious bots can be dealt with and approved bots can be easily controlled. :)

Peace...
 
Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 807,865 other people just like you!

Latest posts

Members online

Top