does anyone have a list of search engine bot IP#'s?

does neone have a good list of the IP#'s of various search engine bots like google, yahoo, msn, etc.. ?

[Join us @ facebook]

There is no such list.

The only way to identify a crawler visit is by the visitor's browser.

what exactly about the browser they use tips you off? i can monitior this with HCL so that is no prob just need 2 know what 2 look for

[Join us @ facebook]

You can check these pages :

http://www.jafsoft.com/searchengines/webbots.html

http://searchenginewatch.com/webmasters/article.php/2167991

method, what will u do with that?

cloaking?

no, when i am watching my visitors with Help Center Live I like to know who's who, but search bots show up as blank so its useless

ummm there was a list compiled a while ago on this site: http://johnny.ihackstuff.com (it's a real slow site, I think the list was in the forums)

I have a list of alot of spider IPs (it's not all spiders, but it's a good start).

http://www.xavierforum.com/files/spider_ips.txt

[Join us @ facebook]

I have a list of alot of spider IPs (it's not all spiders, but it's a good start).

http://www.xavierforum.com/files/spider_ips.txt

Interesting list.

You used some kind of software to populate this?

Well, I looked at my server logs for the last year and that list is the result.

[Join us @ facebook]

Oh I see. You got the difficult way I guess

You can also have a script doing this job.

[.htaccess]

I have a spider trap too, I simply neede to speed up the work a little bit.

This is what my .htaccess looks like:

Code:

RewriteEngine on
RewriteRule ^robots.txt robots.php [L,NC]

Then I have a script called robots.php:

Code:

<?
$server = "localhost"; // MySQL hostname
$username = ""; // MySQL username
$password = ""; // MySQL password
$dbname = ""; // MySQL db name

$db = mysql_connect($server, $username, $password) or die(mysql_error());
mysql_select_db($dbname) or die(mysql_error());

mysql_query("INSERT INTO spiderips VALUES(INET_ATON('". $_SERVER['REMOTE_ADDR'] ."'),'". $_SERVER['HTTP_USER_AGENT'] ."',". time() .")");

mysql_close();

?>
User-agent: *
Disallow: /norobots/

Every time a spider visits my robots.txt I catch the spider. Then I have this code on some other pages (not all of my pages):

Code:

<?$server = "localhost"; // MySQL hostname
$username = ""; // MySQL username
$password = ""; // MySQL password
$dbname = ""; // MySQL db name

$db = mysql_connect($server, $username, $password) or die(mysql_error());
mysql_select_db($dbname) or die(mysql_error());

$spider_info = mysql_query("SELECT * FROM spiderips WHERE ip = INET_ATON('". $_SERVER['REMOTE_ADDR'] ."') OR useragent = '". $_SERVER['HTTP_USER_AGENT'] ."'");
if (mysql_num_rows($spider_info))
{
if (!mysql_query("INSERT INTO spiderips VALUES(INET_ATON('". $_SERVER['REMOTE_ADDR'] ."'),'". $_SERVER['HTTP_USER_AGENT'] ."',". time() .")"))
{
mysql_query("UPDATE spiderips SET useragent = '". $_SERVER['HTTP_USER_AGENT'] ."', lastvisit = '". time() ."' WHERE ip = INET_ATON('". $_SERVER['REMOTE_ADDR'] ."')");
}
}
mysql_close();
?>

This way I get the user agents for the spiders when some of them visits my robots.txt, then I get more IP addresses when I search in the database for the user agent that is a spider. I also get the last time a spider visited one of my sites.

[Join us @ facebook]

That's very clever. Congrats, and thanks for sharing.

BTW you can make this work better if you use the LOW_PRIORITY statement in your inserts and updates

Instead of this :

Code:

$spider_info = mysql_query("SELECT * FROM spiderips WHERE ip = INET_ATON('". $_SERVER['REMOTE_ADDR'] ."') OR useragent = '". $_SERVER['HTTP_USER_AGENT'] ."'");
if (mysql_num_rows($spider_info))
{
if (!mysql_query("INSERT INTO spiderips VALUES(INET_ATON('". $_SERVER['REMOTE_ADDR'] ."'),'". $_SERVER['HTTP_USER_AGENT'] ."',". time() .")"))
{
mysql_query("UPDATE spiderips SET useragent = '". $_SERVER['HTTP_USER_AGENT'] ."', lastvisit = '". time() ."' WHERE ip = INET_ATON('". $_SERVER['REMOTE_ADDR'] ."')");
}
}

you can use this :

Code:

mysql_query("REPLACE LOW_PRIORITY INTO spiderips VALUES(INET_ATON('". $_SERVER['REMOTE_ADDR'] ."'),'". $_SERVER['HTTP_USER_AGENT'] ."',". time() .")")

Cool script, I will use it too.

Thanks for sharing

[Guest]

[37.937]

Googlebot, Yahoo crawler, Msnbot