blocking robots with robots.txt

[Free WordPress Themes]

Hello,

just tried some robots.txt validator and I'm surprised about the result, this is not OK:

Code:

User-agent: Slurp
User-agent: *
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

the validator gives this error:
Line 1    User-agent: Slurp
You specified both the generic user-agent "*" and specific user-agents for this block of code; this could be misinterpreted.

By the way MSN is indexing all the links to this file "/classes/download.php" don't know why...

[Join us @ facebook]

This is the right one (if I understand right what you want to do)

Code:

User-agent: *
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

or

Code:

User-agent: *
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

User-agent: Slurp
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

Both do the same thing

[Free WordPress Themes]

I want to exclude the slurp bot for all pages and for all other the listed file/folders...

[Join us @ facebook]

Then the correct one is :

Code:

User-agent: *
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

User-agent: Slurp
Disallow: *

[Free WordPress Themes]

Then the correct one is :

Code:

User-agent: *
Disallow: /includes/
Disallow: /cache/
Disallow: /classes/download.php
Disallow: /rand_code.php

User-agent: Slurp
Disallow: *

Oeps...

Wrote this a few days ago, maybe it will help.

http://www.helpforwebbeginners.com/webmasters/robot-exclusion-standard.html

PS. Hey Nik, isn't there some way to use link text instead of this giant URL?

[Join us @ facebook]

Wrote this a few days ago, maybe it will help.

http://www.helpforwebbeginners.com/webmasters/robot-exclusion-standard.html

PS. Hey Nik, isn't there some way to use link text instead of this giant URL?

Thanks for the article. You can write [ url = www....... ] anchor [ /url ]

OK, now I've got this working.

Here's the link to the article.

Robot Exclusion Standard

G`day gang 
                I came across this web site optimization tool that does heaps of things to keep your web site free of errors and such. I ran it on one of my web sites and it came up with a "No Archive" tag for the robots. That means my whole site DOES NOT get indexed by googlebot ect. Is that correct? Why on earth would it be there in the first place? The reason given by the optimization tool said it is usually spammers that do that so they dont get caught....i have no clue. I bought the web site premade and NOW i know why i haven`t made a sale on it yet. Slow page loads and over 450 warnings on it     lucky i paid very little for it i guess....lol. Anyone wanting to know about what optimization tool i use or if youknow of one i could check out, please e-mail me or rely to this post.

                            Yikes 

[Join us @ facebook]

Maybe the site is banned from google?

If your robots.txt file is ok you shouldn't have indexing problems with any search engine.

Regarding spammers check this thread

[tr]Maybe the site is banned from google?[/tr]

 Banned? I only just bought it early december. i sure as hell didn`t spam anyone...what do you mean exactly? The website in question is www.kashyhealth.com.au and if you see it, yeah it "looks" ok but as far as making $$$ go...its a HUGE DUD!!! I`ve tried to get things changed on it but it seems they have it setup so i CANNOT do the work myself. I guess if i knew HOW I might be able to but i think they set it up so if any work needs doing, they get it done for (i expect) an outrageous price. So much for the company i bought it off "going to make me rich"...lol. I didnt expect to become a millionaire in 2 months but i sure NEVER expected to have a shoppingcart website and not have a sale after 2months!!! Google did their bit with ADWORDS, burnt badly there, over $600 and no sale. I read that a conversion rate of less than 1% is poor...I don`t even have a conversion rate for that site. Later Guys, I`ve depressed myself now....lol      CYA

                                            Yikes 

[tr]Maybe the site is banned from google?[/tr]    well i did that wrong in my last post!!..lol

                                        Yikes 

[Join us @ facebook]

I don't want to make you feel bad, but these MLM companies are scams. It costed me 4500 euros to realize that, hope you haven't paied them that much....

[Free WordPress Themes]

I don't want to make you feel bad, but these MLM companies are scams. It costed me 4500 euros to realize that, hope you haven't paied them that much....

Nick sometimes is better to ignore a thread

ouch, Yikes I hate seeing people ripped off, send me a pm and I will get one of our boys to do a site review, see if we can find the problem and get you reindexed, no charge

Emmm,

just looking aounds. Im newbie here

[Free WordPress Themes]

Emmm,

just looking aounds. Im newbie here

don't post if you have nothing to say!

[Guest]

[36.918]

Yahoo crawler, Baidu Spider, Googlebot