27, May 2012

Need help on robot.txt - webmaster forum

 
Webdigity webmaster forums
[ Home | Help | Search | Forum's Shop | Archive | Login | Register | Webmaster Directory ]
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization
Topic: Need help on robot.txt		 « previous next »
Pages: [1] Print
Instabuck - The easy way to sell digital products online

Author Topic: Need help on robot.txt  (Read 864 times)
Hunky Junky Monky Man!
**
Posts: 60
408 credits
Members referred : 0
« on: Sep 20, 2010, 04:11:47 pm »
Hi,

Guys i am bit confused regarding the robot.txt .Is there are software to check if they create automatically...?

Thanks & Regards
I feel pretty...
***
Posts: 180
1120 credits
Members referred : 0
« Reply #1 on: Sep 21, 2010, 01:15:15 pm »
I think they are not created automatically... we have to create them manually.
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 5799
46391 credits
Members referred : 3
« Reply #2 on: Sep 21, 2010, 02:23:36 pm »
Just copy paste one Smiley
Trial and Error my two best teachers Cool
Join us @ facebook or twitter

Last blog : Butterfly Marketing 2.0
Cyberpunk Wannabe
*
Posts: 40
266 credits
Members referred : 0
« Reply #3 on: Sep 23, 2010, 10:15:09 am »
you have to go to google webmaster tools they have a inbuilt facility to make a better one for you,,,
Just another rainy day
*
Posts: 1
6 credits
Members referred : 0
« Reply #4 on: Oct 08, 2010, 12:22:30 pm »
Hi all,

I don't think is there any software for creating robots.txt automatically.
Sandwich Artist
*
Gender: Male
Posts: 26
168 credits
Members referred : 0
« Reply #5 on: Oct 08, 2010, 12:43:39 pm »
It's not hard even to understand, and even to do...
Robot.txt instructs crawler which parts of the site should not be indexed...

Why is it important for you?
Because there are parts of the website, which are not part of the completion of the presentation, or there is an old presentation, or do not want the images to be available to the general public, etc. ...

How it works?
Simply type in the notepad a few lines of code and save your file as a robot.txt ...

Example of robot.txt:
Code:
User-agent: *
Disallow: /_workfiles
Disallow: /images
Disallow: /menu
Disallow: /php_uploads
Disallow: /stats
Disallow: /cgi

User agent * - mean - for all user agent
Disallow: folder name (images) or file name (somepage.php) etc...
My name is Bong, James Bong
*
Posts: 10
60 credits
Members referred : 0
« Reply #6 on: Oct 13, 2010, 12:45:31 am »
Nikolas is right on target.  You should just copy and paste one if you have not done so already.  Generally by default they are created to allow all contents to be indexed. If youdo not want portions  of your site indexed then just do a google search and there are plenty of examples out there to follow.
My name is Bong, James Bong
*
Posts: 10
60 credits
Members referred : 0
« Reply #7 on: Oct 19, 2010, 01:01:59 pm »
Its not a Software.This allows all robots to crawl all files.

User-agent: *
Disallow:

* - For All Search engine crawlers

The primary purpose for using a robots.txt file is to gain complete control over the data indexed by the searchbots. Implement a Robots.txt file only when you want to prevent unwanted web pages from being indexed. A robots.txt file is always placed in the root folder of the website where the searchbots can access it easily.
I wish I was an Oscar winner
**
Posts: 98
642 credits
Members referred : 0
« Reply #8 on: Nov 25, 2010, 11:06:44 am »
There are sites that generate the robots.txt for you just google it.
World Wide Whale
***
Posts: 151
948 credits
Members referred : 0
« Reply #9 on: Nov 25, 2010, 02:57:21 pm »
If you don't need something special just copypaste it really
Calm down. It's just a virus.
***
Posts: 171
1090 credits
Members referred : 0
« Reply #10 on: Dec 01, 2010, 03:08:09 pm »
Quote from: Hilarious on Nov 25, 2010, 02:57:21 pm
If you don't need something special just copypaste it really
Agreed, and there are lots of tools in internet to create it
Trackback URI for this entry : http://www.webdigity.com/trackback.php?topic=10643
Tags : seo Bookmark this thread : Digg Del.icio.us Dzone more....

Pages: [1] Print 
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization
Topic: Need help on robot.txt		 « previous next »
Jump to:
User Area
Welcome, Guest. Please login or register.
Did you miss your activation email?
May 27, 2012, 10:17:24 pm





Login with username, password and session length

Donate to our community, and get a permanent link back to your site!

Donate to our community, and get a permanent link back to your site!


Forum Statistics
Total Posts: 62.814
Total Topics: 11.028
Total Members: 21.451
Tutorials : 58
Resources : 929
Designs : 395
Latest Member: sobbin

99 Guests, 1 User online :

11 users online today:

Recent topics
Re: How to
Is the Website Review Secti...
Re: Bing adds features to b...
Re: Google docs, connected ...
Google Chrome Crashes on so...
So.CL Microsoft's new Socia...
Which was your favorite Dis...
Re: Favorite cartoon as a k...

HumanWorks Network
Technology news
Webmaster articles
Sublime web directory
RSS Feed directory and viewer


Web Design Gallery · Whois Lookup · Pagerank · Tag Browsing · Lo-fi version · Syndication · Webmaster forum history · Advertise
Developed by HumanWorks © 2005 - 2012 Webdigity webmaster community · sublime directory
Webdigity Webmaster Forums | Powered by SMF 1.0.12. © 2001-2005, Lewis Media. All Rights Reserved.