AmbrellCooder
Wed 26 May 2010, 12:33 pm GMT +0200
1. Http header tool
2. Robots.txt
3. .htaccess file
I think Google first see http header tool then robots.txt and .htaccess file.
Am i right? Give your useful comment.
sarajuliet
Thu 27 May 2010, 01:31 am GMT +0200
I think the Robots files are getting frequently accessed by the Googlebot, at least this is what I can see after checking my AwStats and Log files online. Although the technology of using some simple text files to control the way search engines crawlers access the servers and websites might be a bit old and outdated, but the frequent visits prove the fast that the Robots files are still important. Although Google prefers to have access to all or at least most parts and folders of the websites but it also tries listening to the words written in the Robots files.
printingblue.com
riedl9176
Fri 30 July 2010, 03:48 am GMT +0200
Page title is not the most at the beginning of the right part
saracriss
Tue 28 September 2010, 05:08 pm GMT +0200
The first thing a crawler see on a website is title.
commodityman
Fri 16 September 2011, 01:44 am GMT +0200
The first thing a crawler see on a website is title.
It is the root domain then Title
gaofan
Sat 24 September 2011, 10:38 am GMT +0200
I think is :
1. .htaccess file
2. Robots.txt
3. Http header tool
deccanbazaar
Sun 29 April 2012, 08:12 am GMT +0200
i never upload robots.txt to any of my website but they are crawled atleast once in a week.
jameseo
Wed 2 May 2012, 01:40 am GMT +0200
I agree with gaofan. I think it's an www rule to view htaccess first to make sure you don't violate any security measures, then robot to not violate any privacy.