Question, Tutorial: Detecting Googlebot with PHP

[Yahoo! Slurp:]

not asking for help on this tutorial, "detecting googlebot with php", just to say that first .

i did follow it, and successfully can detect when googlebot logs into the script so its a great tutorial. however, my question is can the same concept be used for msnbot and yahoo! slurp? i have tried with the following and they don't seem to pick up the visits (when obviously they are crawling)

Yahoo! Slurp:
string for user agent = help.yahoo.com/help/us/ysearch/slurp (found from wiki article on slurp!)

MSNBot/Livebot:
string for user agent = search.live.com (found from webalizer statistics)

i assume that since they are working on webdigity, i must be doing something wrong using the wrong strings or something? here is the PHP code that actually detects them:

Code:

<?php
$spiders = dbquery("SELECT * FROM ".PRE."users WHERE is_spider = '1'");
if (dbrows($spiders)){
while ($bot = dbarray($spiders)){
if (strstr($_SERVER['HTTP_USER_AGENT'], $bot['username']) == true){
$host = gethostbyaddr(USER_IP);
if (substr($host, (strlen($host)-13)) == $bot['location']){
$spider_visited = dbquery("UPDATE LOW_PRIORITY ".PRE."users SET `lastvisit` = '".time()."', `ip` = '".USER_IP."' WHERE `id` = '".$bot['id']."'");
} 
}
}
}
?>

thanks in advance for your help

Yahoo! Slurp

is the yahoo user agent

and

Msnbot

is the one for MSN if I'm not mistaken...

[Join us @ facebook]

Nico is right. Plus you should strpos this because most bots have also a version identifier that changes (eg. Mediabot 1.0)

thanks guys i have updated the details for both bots, waiting on visits now .. and nik, i followed your exact tutorial.. does it show strstr in tutorial? i know why to use strpos, much faster of course thanks for the tip and help as always

this is the code im using now Nik, mind verifying? have changed it a bit

$spiders = dbquery("SELECT * FROM ".PRE."users WHERE is_spider = '1'");
if (dbrows($spiders)){

while ($bot = dbarray($spiders)){

if (strpos($_SERVER['HTTP_USER_AGENT'], $bot['username'])){

//if (strstr($_SERVER['HTTP_USER_AGENT'], $bot['username']) == true){

$host = gethostbyaddr(USER_IP);

if (substr($host, (strlen($host)-13)) == $bot['location']){

$spider_visited = dbquery("UPDATE ".PRE."users SET `lastvisit` = '".time()."' WHERE `id` = '".$bot['id']."'");

if ($bot['ip'] != USER_IP) dbquery("UPDATE LOW_PRIORITY ".PRE."users SET `ip` = '".USER_IP."', `hostname` = '$host' WHERE `id` = '".$bot['id']."'");

} 

}

}
}

[Join us @ facebook]

This:
if (strpos($_SERVER['HTTP_USER_AGENT'], $bot['username']))

should be

if (strpos($_SERVER['HTTP_USER_AGENT'], $bot['username'])!==false)

BTW why do you use the gethostbyaddr() call?

i don't suppose it would be necessary for the spiders, but when users visit the site it checks their IP against what is stored, if it has changed it will update their hostname as well which is stored.. so the hostname call won't come everytime, to save the "every page load" issue.. the reason for storing it for users is for the blacklist, which can ban by IP, email, and hostname

thx for the tip on strpos, ive noticed when not using the 3rd operator "===" or "!==" it operates differently, taken down as a fundamental note thanks again Nikolas!!

still working on MSNbot and Yahoo! Slurp, haven't visited yet

[Original PHP Code that Loops for All Spiders.]

ya.. i dont think those 2 solutions for msn and yahoo are working guys :S.. here is the data its pulling for them, and the evaluated code:

Note: "Data" below is case sensitive.

Original PHP Code that Loops for All Spiders.
//Added for detection of spiders
$spiders = dbquery("SELECT * FROM ".PRE."users WHERE is_spider = '1'");
if (dbrows($spiders)){

while ($bot = dbarray($spiders)){

if (strpos($_SERVER['HTTP_USER_AGENT'], $bot['username'])!==false){

$host = gethostbyaddr(USER_IP);

if (substr($host, (strlen($host)-13)) == $bot['location']){

$spider_visited = dbquery("UPDATE ".PRE."users SET `lastvisit` = '".time()."' WHERE `id` = '".$bot['id']."'");

if ($bot['ip'] != USER_IP) dbquery("UPDATE LOW_PRIORITY ".PRE."users SET `ip` = '".USER_IP."', `hostname` = '$host' WHERE `id` = '".$bot['id']."'");

} 

}

}
}


MSNBot
Data: (name: 'MSNbot'; agent: 'Msnbot')
Code evaluated:
if (strpos($_SERVER['HTTP_USER_AGENT'], "MSNBot")!==false){

$host = gethostbyaddr(USER_IP);

if (substr($host, (strlen($host)-13)) == "Msnbot"){

$spider_visited = dbquery("UPDATE ".PRE."users SET `lastvisit` = '".time()."' WHERE `id` = '".$bot['id']."'");

if ($bot['ip'] != USER_IP) dbquery("UPDATE LOW_PRIORITY ".PRE."users SET `ip` = '".USER_IP."', `hostname` = '$host' WHERE `id` = '".$bot['id']."'");

} 
}


Yahoo! Slurp
Data: name: 'Yahoo! Slurp'; agent: 'Yahoo! Slurp')
Code evaluated:
if (strpos($_SERVER['HTTP_USER_AGENT'], "Yahoo! Slurp")!==false){

$host = gethostbyaddr(USER_IP);

if (substr($host, (strlen($host)-13)) == "Yahoo! Slurp"){

$spider_visited = dbquery("UPDATE ".PRE."users SET `lastvisit` = '".time()."' WHERE `id` = '".$bot['id']."'");

if ($bot['ip'] != USER_IP) dbquery("UPDATE LOW_PRIORITY ".PRE."users SET `ip` = '".USER_IP."', `hostname` = '$host' WHERE `id` = '".$bot['id']."'");

} 
}




with this, MSNBot and yahoo! havent visited yet (According to this script), and i've been watching closely for a few days at my stats, and they are visiting there.

[Join us @ facebook]

Here are a few rules for detecting search engine bots:

if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 6) == "msnbot" )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 20 ) == "Mediapartners-Google"  )//adsense
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 5 ) == "Alexa"  )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 4 ) == "curl"  )
else if ( strstr($_SERVER['HTTP_USER_AGENT'], "Googlebot" ) == true )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 22 ) == "Mozilla/5.0 (Slurp/si;"  )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 12 ) == "Lycos_Spider"  )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 11 ) == "Baiduspider"  )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 17 ) == "Accoona-AI-Agent/" )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 18 ) == "Chitika ContentHit" )
else if ( substr($_SERVER['HTTP_USER_AGENT'], 0, 37 ) == "Mozilla/5.0 (compatible; Yahoo! Slurp" )

are the 2 that say Slurp both for the same Yahoo! Slurp bot?

edit- also i see that it is using strstr for google; that must be why my first version used it

[Join us @ facebook]

The second one is the correct for Yahoo

thanks very much, Nik! this works great for MSNbot and MagpieRSS (so far; Webdigity RSS visiting quite often!)

Yahoo! Slurp hasn't yet, but it surely will soon. Thanks again Nikolas and Webdigity!

[Join us @ facebook]

No worries man. That's what friends are for

[Guest]

[36.301]

Googlebot, Yahoo crawler, Baidu Spider