Headsmacking Tip #13: Don’t Accidentally Block Link Juice with Robots.txt
May 18th, 2009Posted by randfish
A very simple return to the headsmacking series this week (as it’s late here in London and I’ve been up my usual 40+ hours traveling).
We’ve been noticing that a number of websites seeking to block bot access to pages on their domain have been employing robots.txt to do so. While this is certainly a fine practice, the questions we’ve been getting show that there’s a few misunderstandings about what blocking Google/Yahoo!/MSN/other search bots with robots.txt does. Here’s a quick breakdown:
Popularity: unranked [?]