9, July 2008

How can I find out if a site's links are spiderable? - webmaster forum

 
Webdigity webmaster forums
This forum shares its ad revenue with its members!
[ Home | Help | Search | Forum's Shop | Archive | Login | Register | Webmaster Directory ]
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization
Topic: How can I find out if a site's links are spiderable?		 « previous next »
Pages: [1] Print

Author Topic: How can I find out if a site's links are spiderable?  (Read 597 times)
Pinoy Webmaster
.com pimp
*****
Gender: Male
Posts: 1126
6048 credits
Members referred : 0


Philippine Beaches
« on: May 08, 2007, 12:26:04 PM »
is there a site or a tool that can tell me if a site has spiderable links or not?
Philippine Beach Resorts Blog Visit through proxy VBIgnacio Works Visit through proxy Notredame Oilfields Maritime Services Visit through proxy

Last blog : Alona Beach, Bohol, Philippines
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 7946
40629 credits
Members referred : 3
« Reply #1 on: May 08, 2007, 12:28:09 PM »
You don't need a tool for that. If a link does not have the rel="nofollow" attribute it is spiderable.
Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Tim Nash
Global Moderator
Community Supporter ?
Internet Junkie
*****
Posts: 2172
5030 credits
Members referred : 2


Venture Skills - New Media & IT group
« Reply #2 on: May 08, 2007, 12:34:39 PM »
unless its generated by javascript of course Wink
Would you like to be an SEO, let me help with, The Tim Nash introduction to SEO Visit through proxy alternatively for Social media optimisation take a look at the Venture Skills Blog Visit through proxy

Last blog : Its all in the mp3s
Pinoy Webmaster
.com pimp
*****
Gender: Male
Posts: 1126
6048 credits
Members referred : 0


Philippine Beaches
« Reply #3 on: May 08, 2007, 12:42:11 PM »
Code:
If a link does not have the rel="nofollow" attribute it is spiderable.

so i will just check the source code of the site?
Philippine Beach Resorts Blog Visit through proxy VBIgnacio Works Visit through proxy Notredame Oilfields Maritime Services Visit through proxy

Last blog : Alona Beach, Bohol, Philippines
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 7946
40629 credits
Members referred : 3
« Reply #4 on: May 08, 2007, 12:53:39 PM »
Yeah that's enough. If the page you want to check is for some kind of link exchange be sure that this page is linked from the rest of the site (I am getting a lot of link exchange requests from scammers that use non-indexed pages to put their links Smiley )
Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Pinoy Webmaster
.com pimp
*****
Gender: Male
Posts: 1126
6048 credits
Members referred : 0


Philippine Beaches
« Reply #5 on: May 08, 2007, 12:58:59 PM »
how would i know if the link is in a directory disallowed in the robot.txt file? even thou it is not indicated in the actual page, will the links there also not spidered?
Philippine Beach Resorts Blog Visit through proxy VBIgnacio Works Visit through proxy Notredame Oilfields Maritime Services Visit through proxy

Last blog : Alona Beach, Bohol, Philippines
I am a metal monkey!
Administrator
Community Supporter ?
Jedai Sword Master
*****
Gender: Male
Posts: 7946
40629 credits
Members referred : 3
« Reply #6 on: May 08, 2007, 01:27:39 PM »
If the page is linked from somewhere in the site you wont have any problem.

Otherwise you can check it in the site:www...... query in google.
Trial and Error my two best teachers Cool
Join us @ facebook Visit through proxy

Last blog : MIA - Where Nick and Tim
Tim Nash
Global Moderator
Community Supporter ?
Internet Junkie
*****
Posts: 2172
5030 credits
Members referred : 2


Venture Skills - New Media & IT group
« Reply #7 on: May 08, 2007, 01:40:50 PM »
www.mysite.com/robots.txt Visit through proxy - however if the page is indexed then look for no-follows javascript generator androbots-nocontent which is Yahoo new on page stopper
http://ventureskills.wordpress.com/2007/05/02/microformats-for-search-a-real-possibility/ Visit through proxy

for more info
Would you like to be an SEO, let me help with, The Tim Nash introduction to SEO Visit through proxy alternatively for Social media optimisation take a look at the Venture Skills Blog Visit through proxy

Last blog : Its all in the mp3s
Atari ST fan
*
Posts: 7
46 credits
Members referred : 0
« Reply #8 on: Mar 17, 2008, 02:31:25 AM »
Giorgos pointed out in a blog post how there are a variety of ways to limit the crawling of spiders.  Here's the link to his post - http://geoland.org/2007/11/fake-links-nofollow-is-just-the-beginning Visit through proxy

You'd have to look at their robots.txt and/or htaccess.  In addition, they could filter it using javascript to look like a link that would pass juice.

As to a handy way of seeing normal nofollow links, I'd suggest using SEOQuake and set it to line through the nofollow links.  It's available now for IE and Firefox.

Otherwise use the Search Status for Firefox.  It will highlight the nofollow link (annoying to me).
Google dot what?
*
Posts: 2
12 credits
Members referred : 0
« Reply #9 on: Jun 13, 2008, 11:55:39 AM »
Quote from: vbignacio on May 08, 2007, 12:42:11 PM
Code:
If a link does not have the rel="nofollow" attribute it is spiderable.

so i will just check the source code of the site?

Also view the robots.txt of the site. Maybe they disallow the spider to crawl that page/folder.
Trackback URI for this entry : http://www.webdigity.com/trackback.php?topic=6538
Tags : spidered links Bookmark this thread : Digg Del.icio.us Dzone more....

Topic sponsors:
Get a permanent link here for $1.99!


Pages: [1] Print 
Webdigity Webmaster Forums  >  Web site promotion  >  Search Engine Optimization
Topic: How can I find out if a site's links are spiderable?		 « previous next »
Jump to:
User Area
Welcome, Guest. Please login or register.
Did you miss your activation email?
Jul 09, 2008, 04:03:43 PM





Login with username, password and session length

Donate to our community, and get a permanent link back to your site!

Donate to our community, and get a permanent link back to your site!


Forum Statistics
Total Posts: 35.594
Total Topics: 7.356
Total Members: 3.663
Tutorials : 56
Resources : 143
Designs : 220
Latest Member: Stepho

20 Guests, 3 Users online :

18 users online today:

Recent topics
Re: Are you using Squidoo l...
Join webdigity @ facebook...
Re: Your Favorite Car? ...
Re: Submit Free the best to...
hello from Captcha :)...
Re: Looking to Buy a Websit...
Re: Dreamhost - Have I come...
Re: A complete list of keyw...

HumanWorks Network
Technology news
Webmaster articles
Sublime web directory
RSS Feed directory and viewer

Readers

Web Design Gallery · Whois Lookup · Pagerank · Tag Browsing · Lo-fi version · Syndication · Webmaster forum history · Advertise
Developed by HumanWorks © 2005 - 2008 Webdigity webmaster community · sublime directory
Webdigity Webmaster Forums | Powered by SMF 1.0.12. © 2001-2005, Lewis Media. All Rights Reserved.