What does Google see in the website at first time?

[Am i right? Give your useful comment.]

1. Http header tool
2. Robots.txt
3. .htaccess file

I think Google first see http header tool then robots.txt and .htaccess file.

Am i right? Give your useful comment.

I think the Robots files are getting frequently accessed by the Googlebot, at least this is what I can see after checking my AwStats and Log files online. Although the technology of using some simple text files to control the way search engines crawlers access the servers and websites might be a bit old and outdated, but the frequent visits prove the fast that the Robots files are still important. Although Google prefers to have access to all or at least most parts and folders of the websites but it also tries listening to the words written in the Robots files.

printingblue.com

Page title is not the most at the beginning of the right part

The first thing a crawler see on a website is title.

The first thing a crawler see on a website is title.

It is the root domain then Title

I think is :
1. .htaccess file
2. Robots.txt
3. Http header tool

i never upload robots.txt to any of my website but they are crawled atleast once in a week.

I agree with gaofan. I think it's an www rule to view htaccess first to make sure you don't violate any security measures, then robot to not violate any privacy.

[Guest]

[62.814]

Googlebot, bjland5, Yahoo crawler