my care2
make a difference

causes & news

news network

socially conscious news and video shared and rated by the community

Robots.Txt Blocked URLs May Still Show Up In Search Results - Find Out Why !


Science & Tech  (tags: google, robots.txt, url, crawling, search results )

News
- 31 days ago - techrena.net
You have blocked a webpage from crawling by disallowing it in the robots.txt file.But the URL still shows up in the search results and you don't know why this is happening.This page explains why exactly this is happening and what you have to do.
Comments
Or, log in with your
Facebook account:
Please add your comment: (plain text only please. Allowable HTML: <a>)
20
20 log in or sign up to start earning Butterfly Credits today!


Track Comments: Notify me with a personal message when other people comment on this story


Loading Noted By...Please Wait

 

 
Content and comments expressed here are the opinions of Care2 users and not necessarily that of Care2.com or its affiliates.
Copyright © 2009 Care2.com, inc. and its licensors. All rights reserved