9, November 2009

Webdigity tutorials : Detecting GoogleBot with php

 
Webdigity webmaster forums
[ Home | Help | Search | Forum's Shop | Archive | Login | Register | Webmaster Directory ]
Tutorials @ webDigity
Detecting GoogleBot with php

Subscribe to our tutorials feed!

 Recently I came to a major problem with our server. After installing the mod_evasive to our server, the module treated Google bot like a Ddos attacker. The results is that there are thousands of pages in our sites that Google can't crawl, and that way we lose a lot of traffic.

The solution is to find which ips from Google are visiting your sites, and use those ips in the mod_evasive configuration in order to allow them visit your site without time limitations. So if we suppose that we knew all the ips of Google, we would need to use this directive in the Apache configuration file :

Code:
<IfModule mod_evasive20.c>
DOSWhitelist 127.0.0.1 192.168.1.* ........ ips
</IfModule>

But in order to do that we need to know which ips Google is using to crawl our web site. In this you should also know that there are many spam bots that use GoogleBot user agent to visit your site, so we need to check each ip if has a correct reverse dns entry.

Here is the solution to detect those bot visits with PhP :

Code:
<?php
if ( strstr($_SERVER['HTTP_USER_AGENT'], "Googlebot" ) == true ){
  //User has the GoogleBot user agent, but is it a real google bot?
  $host = gethostbyaddr($_SERVER['REMOTE_ADDR']);
  if ( substr($host, (strlen($host)-13)) == 'googlebot.com' )
   //real bot
  else
   //fake bot
}
?>

This snippet is doing the detection work, you can work it a little more if you want to make it log the ips.
Do you like this tutorial?
Feel free to share it with others using Dzone or your favorite bookmarking service. You can see related discussions in our php forum
This tutorial posted by Nikolas from Sublime Directory

Sponsors :

Code library menu :
[ Tutorials home | Submit a tutorial ]

User Area
Welcome, Guest. Please login or register.
Did you miss your activation email?
Nov 09, 2009, 06:41:54 am





Login with username, password and session length

Donate to our community, and get a permanent link back to your site!

Donate to our community, and get a permanent link back to your site!


Forum Statistics
Total Posts: 42.741
Total Topics: 8.375
Total Members: 6.939
Tutorials : 58
Resources : 929
Designs : 341
Latest Member: Auguste Dubuisson

22 Guests, 3 Users online :

15 users online today:

Recent topics
Re: hiii to all of you...
Re: Do you smoke?
Re: What are your favorite ...
Re: google or yahoo?
Re: Your Favorite Car? ...
Re: Yes or no game! Do you ...
Re: what's your favorite fo...
Re: pacquiao or maywheather...

HumanWorks Network
Technology news
Webmaster articles
Sublime web directory
RSS Feed directory and viewer


Web Design Gallery · Whois Lookup · Pagerank · Tag Browsing · Lo-fi version · Syndication · Webmaster forum history · Advertise
Developed by HumanWorks © 2005 - 2009 Webdigity webmaster community · sublime directory
Webdigity Webmaster Forums | Powered by SMF 1.0.12. © 2001-2005, Lewis Media. All Rights Reserved.