robots.txt generator
Choose who may crawl your site, which folders to keep out and where your sitemap is, and copy the robots.txt.
Frequently asked questions
Does blocking AI crawlers hide my site from AI search?
The list here blocks crawlers that collect training data. Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) are separate; blocking them removes your pages from those AI answers too, so they are not in the list.
Is robots.txt a security measure?
No. Well-behaved crawlers follow it, but it is public and anyone can read which folders you list. Protect private pages with a login, not robots.txt.
Where does the file go?
At the root of the domain: https://example.com/robots.txt. Each subdomain needs its own.