Feauture requests

[Promote your blog for free....]

As there are many people that using the script, I thought it would be great to ask them what they need to be puted on the script.

Some things I thought, and I'll put them on later versions are here :

• Abillity to choose the way that the script deals with the spammer ( redirect, deny)
• Multiple .htaccess files parsing. This is allready implemented for the new version, and actually it is detecting the location of the files.
• Script updates. The script will check for new versions when it runs
• I am thinking of creating an additional script that will check the referers for links, in order to be easier to find out who are the spammers
• Better implementation for the users that run php under safe mode

If you have some advices or requests please share them with us

Being able to ban specific useragents for blocking bad robots or by keyword would be nice.
For example, being able to block any domain with *casino* in it would be extremely useful.

I do not think this is possible using the current script but I would think it is possible to add a couple of extra functions to setup.php along with a small menu that would create links to connect to a different update file.

For example, the script currently used HTTP_REFERER and to block bad bots we would need to use HTTP_USER_AGENT and the syntax for the directives would be slightly different.

I am working on this functionality already for aStatSpam which I have converted to a phpNuke content management system module and have so far verified over 2000 bad referers.
My current list can be found at www.code-authors.com/update.php

[Promote your blog for free....]

Well about the first thing you say, it is possible now.

Just put in the custom domains array the word casino and you are done

I do not think this is possible using the current script but I would think it is possible to add a couple of extra functions to setup.php along with a small menu that would create links to connect to a different update file.

That's a nice advice. I will look it when I will start coding the script again.

About the user agent, that's an easy one, I will add it in the future. Untill then you can use this

Cool thanks.
I'll compare your bad useragent list with my own as I have been collecting a few and authenticating them over the last few weeks.

[Promote your blog for free....]

Good. Let us know of your improvements

I am just about to leave for work but here is my current list - I'll reformat these as soon as I have time so they can be copied/pasted into htaccess
anonymiz
asterias
backdoorbot
black hole
blackwidow
blowfish
botalot
builtbottough
bullseye
bunnyslippers
catch
cegbfeieh
charon
cheesebot
cherrypicker
chinaclaw
combine
copyrightcheck
cosmos
crescent
curl
dbrowse
disco
dittospyder
dlman
dnloadmage
download
dreampassport
dts agent
ecatch
eirgrabber
erocrawler
express webpictures
extractorpro
eyenetie
fantombrowser
fantomcrew browser
fileheap
filehound
flashget
foobot
franklin locator
freshdownload
fscrawler
gamespy_arcade
getbot
getright
getweb
go!zilla
go-ahead-got-it
grab
grafula
gsa-crawler
harvest
hloader
hmview
httplib
httpresume
httrack
humanlinks
igetter
image stripper
image sucker
industry program
indy library
infonavirobot
installshield digitalwizard
interget
iria
irvine
iupui research bot
jbh agent
jennybot
jetcar
jobo
joc
kapere
kenjin spider
keyword density
larbin
leechftp
leechget
lexibot
libweb/clshttp
libwww-perl
lightningdownload
lincoln state web browser
linkextractorpro
linkscan/8.1a.unix
linkwalker
lwp-trivial
lwp::simple
mac finder
mata hari
mediasearch
metaproducts
microsoft url control
midown tool
miixpc
missauga locate
missouri college browse
mister pix
moget
mozilla.*newt
mozilla/3.0 (compatible)
mozilla/3.mozilla/2.01
msie 4.0 (win95)
multiblocker browser
mydaemon
mygetright
nabot
navroad
nearsite
net vampire
netants
netmechanic
netpumper
netspider
newsearchengine
nicerspro
ninja
nitro downloader
npbot
octopus
offline explorer
offline navigator
openfind
pagegrabber
papa foto
pavuk
pbrowse
pcbrowser
peval
pompos/
program shareware
propowerbot
prowebwalker
psurf
puf
puxarapido
queryn metasearch
realdownload
reget
repomonkey
rsurf
rumours-agent
sakura
scan4mail
semanticdiscovery
sitesnagger
slysearch
spankbot
spanner
spiderzilla
sq webscanner
stamina
star downloader
steeler
steeler
strip
superbot
superhttp
surfbot
suzuran
swbot
szukacz
takeout
teleport
telesoft
test spider
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
true_robot
tsurf
turing machine
turingos
urlblaze
urlgetfile
urly warning
utilmind
vci
voideye
web image collector
web sucker
webauto
webbandit
webcapture
webcollage
webcopier
webenhancer
webfetch
webgo
webleacher
webmasterworldforumbot
webql
webreaper
website extractor
website quester
webster
webstripper
webwhacker
wep search
wget
whizbang
widow
wildsoft surfer
www-collector-e
www.netwu.com
wwwoffle
xaldon
xenu
zeus
ziggy
zippy

[Promote your blog for free....]

I think that having so much rules in your .htaccess can cause the server run slower

Yes, that is something I had thought about but I found that the page load time was quicker with the directives in htaccess than the page load time I was getting without them due to multiple page loads per useragent.
Before blocking them my average page generation time was over 3 seconds, with the directives in place it is just over 1 second.
I currently have over 2000 blocks on referers, several hundred blocks by IP including some whole country ranges as well as the useragent blocks.
By far the biggest improvement was made by banning 'Slurp' which can consume massive amounts of resources even if you use the robots.txt Crawl-delay directive (crawl delay is specific to Yahoo's slurp / inktomi search bots).

I know I am getting a little off topic now but this information *may* be useful to someone.

[Promote your blog for free....]

That's very interesting.

About the slurp I don't think that is good to ban it, as yahoo can bring some traffic to your site.

You can use this to your robots.txt to make Slurp treat better to your server :

Code:

User-agent: Slurp
Crawl-delay: 12

[Guest]

[34.929]

Yahoo crawler, Googlebot, Baidu Spider