[Iillumin8]

I was reviewing my websites stats and I noticed a folder .... /robots.txt ....
I do not have a folder I created called , /robots.txt , the other folders are all my creations and in my hosts stats clicking on the folder hyper-links to the folder as viewed on the server and net like this one ..... /index.html , but this folder
/robots.txt takes you to the server but ??? I run more than one site on this host but they all have unique url's and are not subdomains the other sites stats also have the same folders ? Is this the servers bots for collating stats? or CIA/NSA/ASIO snooping ,I know that sounds paranoid but the all fairly Anti-Government or is it search engines and if its either of the last 2 HOW did they get themselves as folders on my servers disks ?

[namenotfound]

robots.txt is a text file in which you tell robots what they are allowed to view.

I have this in my robots.txt file:

User-agent: *
Disallow: /images/
Disallow: /logos/

That tells robots (usually referred to as 'bots') not to search in those directories. So if you were to search Yahoo Images or Google Images, you won't find any of MY images because I told those search bots to STAY OUT of those directories because I don't want my images to be indexed

[~Biker~]

Yeah even i have one but i created mine when i started on my site.

User-agent: *
Disallow: /images/
Disallow: /flash/
Disallow: /live/
Disallow: :2222
Disallow: /forums/admin.php
Disallow: /live/livehelp.php?department=1&pingtimes=15
Disallow: /live/livehelp.php?department=2&pingtimes=15
Disallow: /live/livehelp.php?department=3&pingtimes=15
Disallow: /live/livehelp.php?department=4&pingtimes=15
Disallow: /live/livehelp.php?department=5&pingtimes=15

Thats mine

You should also have a .htaccess its some times created if you have front page and stuff.