Q: "If I have some of my site pages blocked in my robots.txt file why do they still show up in Google results?"
A: Robots.txt does not remove your page from the Google Index, it only stops Googlebot from crawling your specified pages. If you have a page showing up in the search results and you apply a robots restriction to that page and that page has no external links, typically that page will disappear from the results. If that page does have external links, the page may continue to show up in the results even after the Robots exclusion is applied. Two easy ways to check that the page is no long being crawled by Googlebot is by looking at your site logs or looking at the search result itself and you will notice that there will no longer be a cache version of that page(there could be other reasons for no cache version). Also, the result will no longer show a description.
The way to 'remove' a page from the index or showing up in the results is to place a noindex meta tag on the page and keep the robots.txt exclusion as well. Another, potentially more dangerous option, is to use the Google URL Removal Request tool in the Google Webmaster Tools for your website where you can remove entire sites, directories, or even specific pages.
Details from Google Webmaster Help Center on Robots.txt and Meta Noindex
Details on Google URL Removal Request from Google Webmaster Central Blog
Thursday, August 16, 2007
My Page Is Blocked With Robots.txt But Still Shows In Results.
There is much confusion on this point.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment