What is robots.txt?

robots.txt is a plain-text file hosted at a site's root — for example https://example.com/robots.txt. It tells automated clients (search engines, scrapers, other bots) which URL paths are allowed or disallowed, and sometimes how fast to crawl.

It is a voluntary convention, not a password wall. Well-behaved bots read it and obey it. Bad actors can ignore it — which is why it is not a security control.

What It Usually Contains:

• User-agent: Which bot the rules apply to (* means everyone)
• Disallow / Allow: Path prefixes the bot should skip or may fetch
• Crawl-delay: A suggested pause between requests (not all bots honor this)
• Sitemap: A hint to where the XML sitemap lives

Example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /products/
Crawl-delay: 2
Sitemap: https://example.com/sitemap.xml

Why Scrapers Should Care:

• Politeness: Honoring robots.txt reduces load and shows good faith
• Blocks: Ignoring it is a fast path to IP bans and legal friction in disputes
• Scope: Disallow does not mean "secret" — those URLs may still be public in a browser — it means "please do not automate here"

What robots.txt Is Not:

• Not encryption — it does not hide content
• Not always law — courts treat it differently by context; still, ignoring it is rarely wise for production scrapers
• Not a substitute for terms of service or privacy rules — those still apply to how you use data

FAQ

If a path is Disallow, can I still scrape it?

Technically many Disallow URLs are still reachable in a browser. For automated collection, treat Disallow as a strong "do not crawl" signal unless you have an explicit agreement with the site.

Does every site have robots.txt?

No. Missing robots.txt usually means no published crawl policy — still throttle yourself and prefer official APIs when available.