Please Help: (.htaccess & robots.txt)

[support.clpafs]

Hi,

I am trying to deny access to our sites from any good/bad bots, If you know of any bots I might have missed please reply.

Also, I am trying to prevent hotlinking in the .htaccess file, If you know of any extensions I might have missed please reply.

ROBOTS.TXT

User-Agent: *
Disallow: /

User-Agent: Adsbot-Google
Disallow: /

User-Agent: Googlebot
Disallow: /

User-Agent: Googlebot-Image
Disallow: /

User-Agent: Googlebot-Mobile
Disallow: /

User-Agent: Mediapartners-Google
Disallow: /

User-Agent: projectplaylist-cache
Disallow: /

User-Agent: projectplaylist
Disallow: /

User-Agent: projectplaylist-directlink
Disallow: /

.HTACCESS

#Ban Bots
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Adsbot-Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Mobile [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-cache [OR]
   RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-directlink
   RewriteRule ^(.*)$ http://go.away/

#Ban Users by IP Addresses
   order allow,deny
   deny from 10.0.0.
   deny from 127.0.
   deny from 192.168.
   allow from all

#Stop Hotlinking
   RewriteEngine On
   RewriteCond %{HTTP_REFERER} !^$
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.com [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.info [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.mobi [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.net [NC]
   RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.org [NC]
   RewriteRule \.(avi|bmp|css|doc|exe|jpg|jpeg|js|gif|ico|mdb|mdi|mdw|midi|mp3|mpg|mpeg|msi|pdf|png|ppt|psd|pub|rtf|swf|wav|wma|xls|xpi|zip)$ - [F]

 

[support.clpafs]

Also, If I deny access to the any files with the extension .txt will robots be able to read the robots.txt file?

 

[tomdkat]

I don't think so.

Peace...