Please Help: (.htaccess & robots.txt)

Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

support.clpafs

Thread Starter
Joined
Apr 7, 2008
Messages
252
Hi,

I am trying to deny access to our sites from any good/bad bots, If you know of any bots I might have missed please reply.

Also, I am trying to prevent hotlinking in the .htaccess file, If you know of any extensions I might have missed please reply.

ROBOTS.TXT
Code:
User-Agent: *
Disallow: /

User-Agent: Adsbot-Google
Disallow: /

User-Agent: Googlebot
Disallow: /

User-Agent: Googlebot-Image
Disallow: /

User-Agent: Googlebot-Mobile
Disallow: /

User-Agent: Mediapartners-Google
Disallow: /

User-Agent: projectplaylist-cache
Disallow: /

User-Agent: projectplaylist
Disallow: /

User-Agent: projectplaylist-directlink
Disallow: /
.HTACCESS
Code:
#Ban Bots
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Adsbot-Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Mobile [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-cache [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-directlink
   RewriteRule ^(.*)$ http://go.away/

#Ban Users by IP Addresses
   order allow,deny
   deny from 10.0.0.
   deny from 127.0.
   deny from 192.168.
   allow from all

#Stop Hotlinking
   RewriteEngine On
   RewriteCond %{HTTP_REFERER} !^$
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.com [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.info [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.mobi [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.net [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.org [NC]
   RewriteRule \.(avi|bmp|css|doc|exe|jpg|jpeg|js|gif|ico|mdb|mdi|mdw|midi|mp3|mpg|mpeg|msi|pdf|png|ppt|psd|pub|rtf|swf|wav|wma|xls|xpi|zip)$ - [F]
 

support.clpafs

Thread Starter
Joined
Apr 7, 2008
Messages
252
Also, If I deny access to the any files with the extension .txt will robots be able to read the robots.txt file?
 
Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 807,865 other people just like you!

Latest posts

Staff online

Top