Apache : Get rid of spam bots

[Join us @ facebook]

After a lot of searching in my logs, I realized that the downtime we had these days was partially due to some spam bots that overloaded the server.

Here is an easy way to get rid of spam bots. Open your .htaccess files and add these lines :

Code:

<IfModule mod_rewrite.c>

  RewriteEngine on
  RewriteBase /
  RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
  RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
  RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
  RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
  RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
  RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
  RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
  RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
  RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
  RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
  RewriteCond %{HTTP_USER_AGENT} (.*)Indy(.*) [OR]
  RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
  RewriteRule ^.* - [F]

</IfModule>

Enjoy

Cool, but what does this actually does?

[Join us @ facebook]

It will deny access to the spam bots.

[Join us @ facebook]

A small addition. You can also put the above line to deny access to users that are trying to leach your site with curl:

Code:

RewriteCond %{HTTP_USER_AGENT} ^Curl(.*) [OR]

most of these things you mention are only necessary for those running their own web host service with multiple servers in a real datacenter, if you use hosts like Website source or Webco planet then this is automatically configured for you.

[Guest]

[36.302]

Googlebot, Yahoo crawler, Gigabot, Baidu Spider