Wednesday, December 28, 2011

URLs restricted by robot.txt on my blogspot site


Are you getting errors/messages in your Google Webmaster Tools account along the lines of URLs Restricted by robot.txt

This is an auto-response for the (very) common issue of blogspot blocking URLs that include /search/ via that robots.txt file.


Do those URLs have the following structure in Google webmaster tools -> Show URLs: Restricted by robots.txt ‎(11)‎


http://www.boxlaeafdipcom/search/label/schema

robottxt_report

No need to woory about this, these are label urls, and Google not crawling label url, Google only crawl Permalink pages url and Archive pages url.

so Google robots restrict all label urls.It blocks the /search/ section of your site from Bots to stop duplication.Otherwise you will get the same content on multiple/many URLs, which could result in strange ranking/search results. So they Automatically Block any URL that has /search/ in it.

For new bloggers, the errors seem to hang aroudn for around some weeks. add to that, if more people start linking to those URLs, then Google bot is going to try to revisit/recrawl, see the robots.txt blocking it, and let you know about it.

You cant do anything for this, as explained above it is intentional and normal, It is not a problem.

0 comments:

Post a Comment

Twitter Delicious Facebook Digg Stumbleupon Favorites More

 
Design by Free WordPress Themes | Bloggerized by Lasantha - Premium Blogger Themes | Web Hosting Coupons