robots.txt

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note ???Please, do not enter??� on an unlocked door e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na??ve to rely on robots.txt to protect it from being indexed and displayed in search results.

robots.txt is related to search engines robots and not specific to PHP projects. it contains instructions for robots that crawl you website such as which urls you dont want to be crawled by the search engines and so on.

Another copy/paste:
http://stackoverflow.com/questions/2911370/what-does-robots-txt-file-do-in-php-project

The primary purpose for using a robots.txt file is to gain complete control over the data indexed by the searchbots. Implement a Robots.txt file only when you want to prevent unwanted web pages from being indexed. A robots.txt file is always placed in the root folder of the website where the searchbots can access it easily.

In that file you tell search engines, which page to index and which not to index.

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note ???Please, do not enter?½ on an unlocked door e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na??ve to rely on robots.txt to protect it from being indexed and displayed in search results.

Yeah, i completely agree

All my friends including almost everything but if you want learn in deep visit: robotstxt dot org

It is not necessary to use robots.txt file if you don't want yo hide any page from crawler.

[Guest]

[62.814]

Googlebot