Referer checker under development

[Join us @ facebook]

As this whole referer spam thing is going more annoying I thought of creating a script that will automatically check the referers of the site if they are spammers or not.

The idea behind this is that I spend about 20-30 minutes every day checking my statistics for spammers.

The new script will store all the referer requests in a mysql database, and when asked, it will validate the links.

That means that there are 3 parts of this script :

1) Store all the referer requests in the database (just developed it)

2) Check the links of the referer. To implement this I will create a small crawler (with Curl or file_get_contents wrap) that will visit the referer page, and will check if the links exists. If not it will store it to the database, and in a text file for later use.

3) Integration with astatspam. The aStatSpam script will finally get this information (from the file that the referer checker will create), and it will parse this info into the .htaccess directives.

How do you see these ideas? Do you have something to add to this?

Any help - feedback will be appreciated

I would love to have this script.

Is there any update on this?

[Join us @ facebook]

Not at the moment.

But I will implement this when I'll have some extra time.

If anyone would like to contribute, please feel free to pm me

That script would be great for my forum too. If you need help on any one thing, just ask.

[Join us @ facebook]

That script would be great for my forum too. If you need help on any one thing, just ask.

Are you familliar with PhP? Maybe I could give you a part of the job

Yeah, I'm not bad with PHP. Just to let you know, databases irritate me, so if at all possible, I wouldn't want to do that.

Anyway, sure, I'd be glad to help.

[Join us @ facebook]

Great, I will pm you tomorrow some stuff.

Thanks

nikolas, what tips you off that certain visitors or posts or whatever it may be, is actually a spammer? besides the obvious giveaways like mass posting.

[Join us @ facebook]

nikolas, what tips you off that certain visitors or posts or whatever it may be, is actually a spammer? besides the obvious giveaways like mass posting.

This subforum is dedicated to referer spam, but I suppose you are talking about the spamming in the forum. The only thing I consider as spam in the forum besides mass posting, is creating threads to advertise services of yours or of your affiliates. After all there is a section on the forum about this, so I will delete (or move there) the inappropriate posts

Great, I will pm you tomorrow some stuff.

Thanks

Still waiting for that PM... I hope you didn't think I abandoned the forums when I left for the holidays.

[Join us @ facebook]

Ooops. I forgot that.

Let me give the code here so, maybe others can help too.

First of all we have a table in the database called referer_check. Here is the SQL to create it :

Code:

CREATE TABLE `referer_check` (
  `referer_url` varchar(255) NOT NULL default '',
  `visit_uri` text NOT NULL,
  `ip` varchar(15) NOT NULL default '',
  `validated` tinyint(1) NOT NULL default '0',
  `is_spammer` tinyint(1) NOT NULL default '0',
  PRIMARY KEY  (`referer_url`)
) TYPE=MyISAM;

Let me explaine the fields above :

referer_url    =>   The url that refered the visitor to our page
visit_uri        =>   The page that visited (full url).
ip                =>   The ip adress of the visitor
validated      =>    It is 1 if we have allready checked this referer
is_spammer  =>    It is 1 if the referer is a spammer

Right now I have developed a small script that can be included from every page of the site, and adds the referers into the table.

Here is what we want to do :

1) A script that will show all the urls in a list, the user will check the desired ones, and then he/she will push a 'Check referers' button which will evaluate all the referers. To do this the script will have to download the page (CURL or file_get_contents or both ) and then check if the link to the `visit_url` exists (preg or strpos)

2) The script must have some kind of flexibillity. Eg. the user must be able to list the referers using various variables, eg. show only non validated, show only non spammers, alphabetically, etc.

3) The script must have password authentication. So the user will have to put his username/password before doing anything.

4) There must be a layout for this administration area.

5) All the configuration variables will be stored in ini files. I have allready sone this so for everything you code just use dummy variables, and I will change your code later to integrate it with the ini parse class.

Well that's all. If anyone can do some of this, please reply here so we can all work on different areas, and make the development of the script faster.

referer_url    =>   The url that refered the visitor to our page
visit_uri        =>   The page that visited (full url).
ip                =>   The ip adress of the visitor
validated      =>    It is 1 if we have allready checked this referer
is_spammer  =>    It is 1 if the referer is a spammer

I have no idea how to do the first part, I have 0 experience in making referal scripts, so I don't know how to make the URL. I imagine it would involve apache and the user's username though.

The page visited seems easy enough with cookies, but if they user blocks cookies...

The IP address of the visitor should be very easy. Just use the PHP function to get the user's IP, then log it in a MySQL database for further usage.

Now, the validated and spammer parts will be the easiest, but the hardest to use. I suggest you just make a PHP checkbox form saying yea or nay for if the user is valid. Of course, this would have to placed in the admin panel of SMF, which I assume you can do because of all the stuff you integrated to this forum.

I'm not sure how much I can help overall, but just ask me little questions and I'll try and piece the answers together.

[Join us @ facebook]

Ok.

The part that inserts the data into the database is done.

Also I have some seperate functions (from other projects) that can download and analyze the page if it is a spammer or not.

Something that would be very helpfull is an html layout for the control panel. Can you make this?

Sure, I can probably do that. Just tell me what you want, for example, what links to have, what colors, layout style, etc.

[Join us @ facebook]

I really don't care about the colors.

I just want a css layout that will display the admin page better.

The links will be :

Configuration

Validate Referers

Create .htaccess

Support forum

Alright, I'll get to work on it.

[Join us @ facebook]

OK. Thanks, I will wait for your work.

Anyone else wants to contribute?

[Guest]

[35.590]

Googlebot, Yahoo crawler, Nikolas, Baidu Spider