5, September 2008

list of all google crawlers? - webmaster forum

 
Webdigity webmaster forums
This forum shares its ad revenue with its members!
[ Home | Help | Search | Forum's Shop | Archive | Login | Register | Webmaster Directory ]
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization  >  Google SEO
Topic: list of all google crawlers?
« previous next »
Pages: [1] Print

Author Topic: list of all google crawlers?  (Read 1934 times)
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« on: Jul 23, 2006, 02:09:01 PM »

Hello,

does someone know a complete list of ranges/ip addresses used by the google webcrawler?

and maybe the same list for MSN?


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #1 on: Jul 23, 2006, 02:12:38 PM »

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #2 on: Jul 23, 2006, 02:16:08 PM »

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

just noticed that this range is from google:
NetRange:   66.249.64.0 - 66.249.95.255

but what is a better way to identify the search engine in your stats the "HTTP_USER_AGENT" ?


Last blog : Is your website is down? Know before your visitors do!
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #3 on: Jul 23, 2006, 02:16:54 PM »

Nick, why is thread about seo?


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #4 on: Jul 23, 2006, 02:21:24 PM »

Nick, why is thread about seo?

I guess because crawlers about SEO Smiley

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

just noticed that this range is from google:
NetRange:   66.249.64.0 - 66.249.95.255

but what is a better way to identify the search engine in your stats the "HTTP_USER_AGENT" ?


I think this range is from Google ( I remember the 66 in the start), but rather than user agent there is not other good way.

But you can allways check some things. For example google fetches robots.txt anytime it visits a site (at least once per session), also it is visiting only one page per time, and it never sends referer headers.

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #5 on: Jul 23, 2006, 02:24:57 PM »

I need this test for something other, I need a feature to block users to access a page only a limited number of time and I wnat to do that with a test on the ip address (maybe also a cookie). That why I need a check to know who is google. I have hundreds of visits with aa fake "HTTP_USER_AGENT"


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #6 on: Jul 23, 2006, 02:30:57 PM »

Check mod_limitipconn Visit through proxy. It does all the work automatically Smiley

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #7 on: Jul 23, 2006, 02:33:21 PM »

Check mod_limitipconn Visit through proxy. It does all the work automatically Smiley

can I access the module via php?


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #8 on: Jul 23, 2006, 02:42:02 PM »

Check mod_limitipconn Visit through proxy. It does all the work automatically Smiley

can I access the module via php?

No you just set some directives in virtual host or in htaccess.

Here is an example :

Code:
<IfModule mod_limitipconn.c>
    <Location />
        MaxConnPerIP 7
        # exempting images from the connection limit is often a good
        # idea if your web page has lots of inline images, since these
        # pages often generate a flurry of concurrent image requests
        NoIPLimit images/* css/* admin/* js/*
    </Location>
  </IfModule>

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #9 on: Jul 23, 2006, 02:49:52 PM »

This is a nice feature, but I have to prevent users from using a form to often (because they are searchiung for advertisers), that will say I hev to protect only two result lists from a complete website...


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #10 on: Jul 23, 2006, 02:54:37 PM »

If those files exist in the same directory then you can for sure do this with this mod.

Otherwise I am not sure if you can manage it to check only two files instead of the whole site.

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #11 on: Jul 23, 2006, 03:50:49 PM »

If those files exist in the same directory then you can for sure do this with this mod.

Otherwise I am not sure if you can manage it to check only two files instead of the whole site.
I will remember this mod, but for now I need to do something else, thanks


Last blog : Is your website is down? Know before your visitors do!
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #12 on: Jul 23, 2006, 03:51:50 PM »

buy the way how does it work if the visitor is surfing via a proxy server?


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #13 on: Jul 23, 2006, 03:53:43 PM »

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google Smiley )

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Global Moderator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 6309
38674 credits
Members referred : 374


It's time to use PHP5!


« Reply #14 on: Jul 23, 2006, 03:55:02 PM »

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google Smiley )

I remember that modem users having 3 diff. ip addresses within 5 minutes... what about them?


Last blog : Is your website is down? Know before your visitors do!
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 8037
41179 credits
Members referred : 3



« Reply #15 on: Jul 23, 2006, 04:08:26 PM »

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google Smiley )

I remember that modem users having 3 diff. ip addresses within 5 minutes... what about them?

I didn't know of this. Dial up users have one ip, and it changes whenever the connection resets.

If someone has multiple ips, this mod can't do anything as it works only with the ip (no sessions involved)

Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Trackback URI for this entry : http://www.webdigity.com/trackback.php?topic=3378
Tags : google seo proxy robots.txt Bookmark this thread : Digg Del.icio.us Dzone more....

Topic sponsors:
Get a permanent link here for $1.99!


Pages: [1] Print 
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization  >  Google SEO
Topic: list of all google crawlers?
« previous next »
Jump to:
User Area
Welcome, Guest. Please login or register.
Did you miss your activation email?
Sep 05, 2008, 02:15:18 PM





Login with username, password and session length

Donate to our community, and get a permanent link back to your site!

Donate to our community, and get a permanent link back to your site!


Forum Statistics
Total Posts: 36.293
Total Topics: 7.476
Total Members: 3.897
Tutorials : 56
Resources : 143
Designs : 220
Latest Member: FlorianHofmann

20 Guests, 4 Users online :

15 users online today:



Readers

Web Design Gallery · Whois Lookup · Pagerank · Tag Browsing · Lo-fi version · Syndication · Webmaster forum history · Advertise
Developed by HumanWorks © 2005 - 2008 Webdigity webmaster community · sublime directory
Webdigity Webmaster Forums | Powered by SMF 1.0.12. © 2001-2005, Lewis Media. All Rights Reserved.