In plain English
It's the sign at a property's entrance saying which areas are open to visitors. Well-behaved visitors follow it; it doesn't lock any doors.
Used in a sentence
“Robots.txt keeps crawlers out of the admin area and points them to the sitemap.”
“One line in our robots.txt was blocking Google from the entire site for months.”
In detail
The file lives at yourdomain.com/robots.txt. It lists rules for crawlers, such as which folders to skip, and usually where to find the sitemap. Reputable search engines follow it, but it isn't security: anyone can still visit those addresses, and the file itself is public.
A blocked page can still appear in search results if other sites link to it, just without a description, because the crawler couldn't read it. That's why robots.txt is for managing crawling, not for hiding pages. To keep a page out of results, use noindex instead.
Why it matters for your business
A single wrong line in robots.txt can hide a whole site from Google, so it's worth checking.
- Points crawlers to your sitemap
- Keeps them out of admin areas
- Not a way to hide pages
Easy to mix up
Robots.txt isn't noindex. Robots.txt stops crawling; noindex stops a page appearing in results. Blocking a noindexed page in robots.txt can stop search engines ever seeing the noindex.
Related terms
Crawling, Noindex and Sitemap. Or browse the whole glossary.
