Congratulations to AcaCandy on her 100,000th post!
There's no such thing as a stupid question, but they're the easiest to answer.
JoinTour
Login
 
Tag Cloud
acer black screen blue screen boot bsod computer connection crash css dell driver drivers email error ethernet excel firefox firefox 3 hard drive internet internet explorer itunes laptop linux malware monitor motherboard network networking outlook outlook 2003 outlook 2007 outlook express partition password printer problem router slow software sound spyware trojan usb video virus vista windows windows xp wireless
Web Design & Development
Search
Search in:
 
Advanced Search
Tech Support Guy Forums > Internet & Networking > Web Design & Development >
Solved: What is---> /robots.txt


HELLO AND WELCOME! Before you can post your question, you'll have to register -- it's completely free! Click here to join today! We highly recommend that you print a copy of our Guide for New Members. Enjoy!

Closed Thread
 
Thread Tools
Iillumin8's Avatar
Registered User with 24 posts.
 
Join Date: Nov 2005
Experience: Trying
21-Feb-2006, 03:05 AM #1
Question Solved: What is---> /robots.txt
I was reviewing my websites stats and I noticed a folder .... /robots.txt ....
I do not have a folder I created called , /robots.txt , the other folders are all my creations and in my hosts stats clicking on the folder hyper-links to the folder as viewed on the server and net like this one ..... /index.html , but this folder
/robots.txt takes you to the server but ??? I run more than one site on this host but they all have unique url's and are not subdomains the other sites stats also have the same folders ? Is this the servers bots for collating stats? or CIA/NSA/ASIO snooping ,I know that sounds paranoid but the all fairly Anti-Government or is it search engines and if its either of the last 2 HOW did they get themselves as folders on my servers disks ?
namenotfound's Avatar
Computer Specs
Distinguished Member with 2,188 posts.
 
Join Date: Apr 2005
Location: New York
Experience: I know what I know, I am
21-Feb-2006, 12:09 PM #2
robots.txt is a text file in which you tell robots what they are allowed to view.

I have this in my robots.txt file:

Code:
User-agent: *
Disallow: /images/
Disallow: /logos/
That tells robots (usually referred to as 'bots') not to search in those directories. So if you were to search Yahoo Images or Google Images, you won't find any of MY images because I told those search bots to STAY OUT of those directories because I don't want my images to be indexed
__________________
-----------------------------
| 404: Name Not Found |
-----------------------------
PLEASE NOTE: If I happen to help you in a post, or just simply reply to it, doesn't mean I want to be bombarded with PMs. I answer all questions in posts, not in PMs. Thank you, and have a good day.

<?php $h = 'Hello '; $w = 'World'; echo $h.$w; ?>

My Favorite Editors:
Windows: Crimson Editor
Mac: Taco HTML Edit
Linux: gPHPEdit
~Biker~'s Avatar
Member with 62 posts.
 
Join Date: Dec 2005
Experience: eh?
21-Feb-2006, 01:50 PM #3
Yeah even i have one but i created mine when i started on my site.

Code:
User-agent: *
Disallow: /images/
Disallow: /flash/
Disallow: /live/
Disallow: :2222
Disallow: /forums/admin.php
Disallow: /live/livehelp.php?department=1&pingtimes=15
Disallow: /live/livehelp.php?department=2&pingtimes=15
Disallow: /live/livehelp.php?department=3&pingtimes=15
Disallow: /live/livehelp.php?department=4&pingtimes=15
Disallow: /live/livehelp.php?department=5&pingtimes=15
Thats mine

You should also have a .htaccess its some times created if you have front page and stuff.
Closed Thread

THIS THREAD HAS EXPIRED.
Are you having the same problem? We have volunteers ready to answer your question, but first you'll have to join for free. Need help getting started? Check out our Welcome Guide.


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
WELCOME TO TECH SUPPORT GUY! Are you looking for the solution to your computer problem? Join our site today to ask your question -- for free! Our site is run completely by volunteers who help people like you solve computer problems. See our Welcome Guide to get started.



Thread Tools


You Are Using:
Server ID
Advertisements do not imply our endorsement of that product or service.
All times are GMT -4. The time now is 11:11 PM.
Copyright © 1996 - 2008 TechGuy, Inc. All rights reserved.
Powered by vBulletin, Copyright © 2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.1.0
Powered by Cermak Technologies, Inc.