Robots.txt - Config format

Robots.txt is a config format created in 1994 by Martijn Koster.

#2320on PLDB

30Years Old

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.

Tags: configFormat
Early development of Robots.txt happened in https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html
Read more about Robots.txt on the web: 1. 2.

Example from the web:

User-agent: googlebot        # all Google services
Disallow: /private/          # disallow this directory

User-agent: googlebot-news   # only the news service
Disallow: /                  # disallow everything

User-agent: *                # any robot
Disallow: /something/        # disallow this directory

Language features

Feature	Supported	Example	Token
Comments	✓	# A comment
Line Comments	✓	# A comment	#