list of all google crawlers?

Hello,

does someone know a complete list of ranges/ip addresses used by the google webcrawler?

and maybe the same list for MSN?

[Join us @ facebook]

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

just noticed that this range is from google:
NetRange:   66.249.64.0 - 66.249.95.255

but what is a better way to identify the search engine in your stats the "HTTP_USER_AGENT" ?

Nick, why is thread about seo?

[Join us @ facebook]

Nick, why is thread about seo?

I guess because crawlers about SEO

I think there was a thread about this in the forum, but anyway it is impossible to have a complete list while Google has too many datacenters/ips

just noticed that this range is from google:
NetRange:   66.249.64.0 - 66.249.95.255

but what is a better way to identify the search engine in your stats the "HTTP_USER_AGENT" ?

I think this range is from Google ( I remember the 66 in the start), but rather than user agent there is not other good way.

But you can allways check some things. For example google fetches robots.txt anytime it visits a site (at least once per session), also it is visiting only one page per time, and it never sends referer headers.

I need this test for something other, I need a feature to block users to access a page only a limited number of time and I wnat to do that with a test on the ip address (maybe also a cookie). That why I need a check to know who is google. I have hundreds of visits with aa fake "HTTP_USER_AGENT"

[Join us @ facebook]

Check mod_limitipconn. It does all the work automatically

Check mod_limitipconn. It does all the work automatically

can I access the module via php?

[Join us @ facebook]

Check mod_limitipconn. It does all the work automatically

can I access the module via php?

No you just set some directives in virtual host or in htaccess.

Here is an example :

Code:

<IfModule mod_limitipconn.c>
    <Location />
        MaxConnPerIP 7
        # exempting images from the connection limit is often a good
        # idea if your web page has lots of inline images, since these
        # pages often generate a flurry of concurrent image requests
        NoIPLimit images/* css/* admin/* js/*
    </Location>
  </IfModule>

This is a nice feature, but I have to prevent users from using a form to often (because they are searchiung for advertisers), that will say I hev to protect only two result lists from a complete website...

[Join us @ facebook]

If those files exist in the same directory then you can for sure do this with this mod.

Otherwise I am not sure if you can manage it to check only two files instead of the whole site.

If those files exist in the same directory then you can for sure do this with this mod.

Otherwise I am not sure if you can manage it to check only two files instead of the whole site.

I will remember this mod, but for now I need to do something else, thanks

buy the way how does it work if the visitor is surfing via a proxy server?

[Join us @ facebook]

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google )

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google )

I remember that modem users having 3 diff. ip addresses within 5 minutes... what about them?

[Join us @ facebook]

buy the way how does it work if the visitor is surfing via a proxy server?

There's no problem, as the proxy has one ip adress. The only possible problem could be if there were many visitors using the same proxy at the same time, but this is almost impossible (at least for a site with less traffic than Google )

I remember that modem users having 3 diff. ip addresses within 5 minutes... what about them?

I didn't know of this. Dial up users have one ip, and it changes whenever the connection resets.

If someone has multiple ips, this mod can't do anything as it works only with the ip (no sessions involved)

[Guest]

[62.865]

Googlebot, Msnbot, Markirosyan, doris421pu, Yahoo crawler