1. Computer problem? Tech Support Guy is completely free -- paid for by advertisers and donations. Click here to join today! If you're new to Tech Support Guy, we highly recommend that you visit our Guide for New Members.

Please Help: (.htaccess & robots.txt)

Discussion in 'Web Design & Development' started by support.clpafs, Sep 30, 2008.

Thread Status:
Not open for further replies.
  1. support.clpafs

    support.clpafs Thread Starter

    Joined:
    Apr 7, 2008
    Messages:
    252
    Hi,

    I am trying to deny access to our sites from any good/bad bots, If you know of any bots I might have missed please reply.

    Also, I am trying to prevent hotlinking in the .htaccess file, If you know of any extensions I might have missed please reply.

    ROBOTS.TXT
    Code:
    User-Agent: *
    Disallow: /
    
    User-Agent: Adsbot-Google
    Disallow: /
    
    User-Agent: Googlebot
    Disallow: /
    
    User-Agent: Googlebot-Image
    Disallow: /
    
    User-Agent: Googlebot-Mobile
    Disallow: /
    
    User-Agent: Mediapartners-Google
    Disallow: /
    
    User-Agent: projectplaylist-cache
    Disallow: /
    
    User-Agent: projectplaylist
    Disallow: /
    
    User-Agent: projectplaylist-directlink
    Disallow: /
    .HTACCESS
    Code:
    #Ban Bots
       RewriteEngine on
       RewriteCond %{HTTP_USER_AGENT} ^Adsbot-Google [OR]
       RewriteCond %{HTTP_USER_AGENT} ^Google [OR]
       RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image [OR]
       RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Mobile [OR]
       RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [OR]
       RewriteCond %{HTTP_USER_AGENT} ^projectplaylist [OR]
       RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-cache [OR]
       RewriteCond %{HTTP_USER_AGENT} ^projectplaylist-directlink
       RewriteRule ^(.*)$ http://go.away/
    
    #Ban Users by IP Addresses
       order allow,deny
       deny from 10.0.0.
       deny from 127.0.
       deny from 192.168.
       allow from all
    
    #Stop Hotlinking
       RewriteEngine On
       RewriteCond %{HTTP_REFERER} !^$
       RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.com [NC]
       RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.info [NC]
       RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.mobi [NC]
       RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.net [NC]
       RewriteCond %{HTTP_REFERER} !^http(s)?://(.*\.)?example.org [NC]
       RewriteRule \.(avi|bmp|css|doc|exe|jpg|jpeg|js|gif|ico|mdb|mdi|mdw|midi|mp3|mpg|mpeg|msi|pdf|png|ppt|psd|pub|rtf|swf|wav|wma|xls|xpi|zip)$ - [F]
     
  2. support.clpafs

    support.clpafs Thread Starter

    Joined:
    Apr 7, 2008
    Messages:
    252
    Also, If I deny access to the any files with the extension .txt will robots be able to read the robots.txt file?
     
  3. tomdkat

    tomdkat Retired Trusted Advisor

    Joined:
    May 6, 2006
    Messages:
    7,148
    I don't think so.

    Peace...
     
As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 733,556 other people just like you!

Loading...
Thread Status:
Not open for further replies.

Short URL to this thread: https://techguy.org/754924

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice