Bots that slow your server down, any thing you can do?

I know getting your server indexed in a search engine is a good thing. But is there anything you can do when you have a ton of bots on your different website's and it starts slowing the server down?

[Join us @ facebook]

I am not sure if you can really do something for this, except maybe using some kind of caching.

[SEO Voter]

Maybe try editting your robots.txt file to limit the bots to certain areas. There is also a meta tag in the head of your html that instructs bots to only visit after a certain period. I think though that the first one would work better than the latter...

[Join us @ facebook]

Yes the revisit-after meta. I use it, but many bots are not follow this rule

[SEO Voter]

I thought they wouldn't...

I sm sure that the bots will slow down eventually.

Try robots.txt, and if that doesn't work, just ban their IPs

It very much depends on what you call a 'bot' - many are simply image or script harvesters which server no useful purpose at all it terms of Ranking PR etc.
The worst bot I know of for using up server resources is the Inktomi/Slurp bot used by Yahoo.
Things can be so bad with this bot that it has its own meta function you can use in robots.txt called Crawl-Delay

[Join us @ facebook]

Thanks for sharing that Guardian. I was aware of this directive.

Here is how it works :

Code:

User-agent: Slurp
Crawl-delay: 10

Mine is set to 60 as I regularly get between 100 to 200 of them constantly throughout the day.
Now, here is a scary thought.........
Many servers are configured incorreclty and allow the traversal of non-existant directories for example try on a site www.mysite.com/index.php/index.php
Many sites that will work on.
Yahoo will actually try to spider for non existant url's to check its bot is working. In th cases where these server configurations allow traversal of non existant url's the header response to the bot will be  a 200 (instead of 404) so the non existant url will actually get indexed !!!

[SEO Voter]

that could be a problem especially if you have mod-rewrites pages that are not checked, cause in some cases you may even have any number of pages that have the exact same content and the may think that you are spamming!

Exactly.
As as there is no way to know what url Slurp or other bots may choose to test a server for correct header response, so you cannot exactly 'anticipate' what it is and set up a redirect or redirectmatch.

As the server actualy produces a page, it has content and therefore can be indexed even though the url does not actually exist.
The index.php/index.php is a classic example of a non existant url that I have seen indexed - Google search gives some excellent examples..

[Join us @ facebook]

Google search gives some excellent examples..

650.000 indexed pages. That's a lot I guess.

I have seen that even with pages that have 404 error codes, Slurp is visiting them again and again. I had made a error in a link before about two months, and even now Slurp is still visits that non existed page.

Maybe this is a bug of the Slurp crawler or somthing.

[SEO Voter]

I have added a perminant redirect to my pages that are incorrectly indexed or moved pages, the they have been visiting them for months still... don't know why they are waisting their time?

I have added a perminant redirect to my pages that are incorrectly indexed or moved pages, the they have been visiting them for months still... don't know why they are waisting their time?

If you are redirecting, then the bot(s) see it as a valid page as the header response is 200
If you see a bot crawling an invalid url by examining your server logs etc, you should redirect to a 404 error page so the bot recieves the correct header response and will eventually drop the url from the indexed cache.
Of course, sometimes it is preferable to redirect to a valid page but eventually you end up in a situation where a bot may thnik you are spamming due to duplicate content - this is a nightmare lol

I don't think that having duplicate content is a really big problem for the search engines.

If you have duplicate pages, they just don't get indexed.

MODs : I think this post should be in the SEO category.

[Join us @ facebook]

Topic moved. Thanks

As for the duplicate content, I think that you may be right but I can't be sure about this at this moment.

You can get uh, reprimanded or whatever by Google if you have duplicate content.

Anyway, I don't really have a problem with bots, so I can't help. My server's big and fast enough to handle it.

~Crystal

[SEO Voter]

I have added a perminant redirect to my pages that are incorrectly indexed or moved pages, the they have been visiting them for months still... don't know why they are waisting their time?

If you are redirecting, then the bot(s) see it as a valid page as the header response is 200
If you see a bot crawling an invalid url by examining your server logs etc, you should redirect to a 404 error page so the bot recieves the correct header response and will eventually drop the url from the indexed cache.
Of course, sometimes it is preferable to redirect to a valid page but eventually you end up in a situation where a bot may thnik you are spamming due to duplicate content - this is a nightmare lol

I use the following to show the surps that it is permanent:

Code:

Header( "HTTP/1.1 301 Moved Permanently" );

but this is going a bit of topic...

[Guest]

[38.666]

Yahoo crawler, Googlebot, Msnbot, Gigabot