Web Design Dictionary - A free glossary of web site design terms.
Web Design Dictionary


  The robots.txt file is a file in the root directory of a web site that controls spider access to the site. It can allow or disallow specific spiders access to specific pages on the site. Spiders can ignore the robots.txt file, but the reputable ones check it first and index only the pages that the robots.txt file allows them to. Notably, all of the major search engines and the Wayback Machine honor the robots.txt file.

Last edited by Andre of http://www.k9dir.com
Permalink to this entry



GoDaddy.com Deal of the Week! - 30% off your order! Offer expires 8/14/12.

GoDaddy.com Hosting just $1.99/mo! 120x60  


Dictionary Home Add A New Term Find a Web Site DesignerBuy a LogoAbout / ContactLogin Disclaimer

© Copyright 2010 - 2018, WebDesignDictionary.com. All Rights Reserved.
Web Design Dictionary - A glossary of web site design terms
WebDesignDictionary.com is a division of DIR Creative.
Admin Login

God First