$ robots --test
Dev and APIrobots.txt Tester — See Which Crawlers Can Reach Each URL
Find out what a site’s robots.txt allows, test a URL against every crawler — search and AI — and see the mistakes costing you crawl budget.
- no sign-up
- free
- nothing stored
What robots.txt is
robots.txt is a plain text file that always lives in the same place: the root of the domain, at example.com/robots.txt. It is the first thing a crawler fetches when it arrives at a site, before any page. The content is a list of rules grouped by crawler — each block opens with a User-agent: and is followed by Allow: and Disallow: directives that open or close paths.
It is worth being clear about what the file is not: it blocks nothing technically. A crawler that decides to ignore it reaches the site just the same, because the server keeps answering normally. robots.txt works by convention — the crawlers that matter to your traffic (Google, Bing, and now the AI ones) honour it because ignoring it would cost them reputation. For content that genuinely must stay out of reach, the instrument is server-side authentication, not a line in a public file.
How to test whether a URL is blocked
Reading a robots.txt by eye is misleading. The rule that applies to a crawler comes from the most specific group matching its name, and only that one — if a Googlebot block exists, Googlebot ignores the * block entirely, including restrictions that live only there. Within the chosen group, the longest matching path wins, and a tie between Allow and Disallow is resolved in favour of Allow.
That is what the URL field on this page is for. You give it a path on the site and the tool resolves that precedence chain for each crawler separately, showing who gets in and who stays out. The result often surprises: a Disallow placed in the wildcard group frequently does not apply to Googlebot, which has its own block a few lines above.
AI crawlers: GPTBot, ClaudeBot and the rest
Until recently, deciding about robots.txt meant deciding about search engines. There is now a second category, and it splits into two purposes worth keeping apart. Some crawlers collect content to train models — GPTBot, CCBot, Google-Extended, Applebot-Extended. Others fetch pages at the moment someone asks a question, to cite the answer with a link: OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot.
The distinction matters because the decision differs for each. Blocking the training crawlers is a choice about content ownership. Blocking the retrieval ones means giving up on being cited when someone asks an assistant about your subject — the 2026 equivalent of requesting removal from Google. One specific trap: Google-Extended controls whether your content trains Gemini, but it does not take your site out of AI Overviews in search results. No separate control exists for that.
The mistakes that cost the most
The first is a forgotten Disallow: /. It blocks the entire site and is almost always left over from a staging environment promoted to production with the file attached. The classic symptom is organic traffic falling off a cliff a few days after a deploy, with no content change to explain it.
The second is blocking a page in robots.txt that carries noindex. It looks like reinforcement and is the opposite: a crawler cannot read the meta tag of a page it is forbidden to download, so the page never leaves the index. If the intent is removal from search, crawling has to stay open.
The third is quieter: blocking the site’s CSS and JavaScript. Search engines render pages the way a browser would, and without the layout resources their assessment of responsiveness and visible content comes out wrong. Beyond these, the tool above flags the rest of the repertoire — paths missing a leading slash, rules declared before any User-agent, a Crawl-delay high enough to make crawling a large site impractical, and the case where the server returns an HTML error page instead of the file, with status 200, which makes every crawler treat robots.txt as absent.
Privacy in this tool
Another domain’s robots.txt cannot be fetched by your browser — the CORS policy prevents it — so the request goes out from our server with an identifiable user agent (CodeCrushBot). The address you type is used for that lookup and nothing is stored: no database, no application log, and the response is marked Cache-Control: no-store. The file being fetched is public by definition — anyone can open it in a browser.
## faq
Frequently asked questions
What is robots.txt?
It is a plain text file at the root of a site (example.com/robots.txt) telling automated crawlers which parts of the site they may walk through. It follows RFC 9309 and works by convention, not by technical enforcement: each block declares a "User-agent" (the crawler a rule applies to) followed by "Allow" and "Disallow" directives with paths. Well-behaved crawlers — Googlebot, Bingbot, GPTBot — read the file before anything else and honour what it says. A malicious scraper simply ignores it, and the file has no way to stop it.
Does blocking a page in robots.txt remove it from Google?
No, and this is the most expensive misunderstanding in technical SEO. "Disallow" prevents crawling, not indexing. If other sites link to the blocked URL, Google may index it anyway — it just cannot read the content, so the result shows up with "no information is available for this page". Worse: if you block a page in robots.txt that already carries a noindex meta tag, Google will never be able to read that tag, and the page stays stuck in the index. To remove something from search, use noindex and leave crawling open.
How do I block GPTBot, ClaudeBot and other AI crawlers?
Create a group for each one in robots.txt. For example: "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI’s crawler used for model training. The agents that matter today are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot, CCBot (Common Crawl, which feeds several models), Bytespider (ByteDance), Applebot-Extended and meta-externalagent. One caveat about Google-Extended: it controls whether your content trains Gemini, but it does not remove your site from AI Overviews in search results — there is no separate control for that.
What is the difference between Disallow and noindex?
They act at different stages. "Disallow" in robots.txt acts before the visit: the crawler never downloads the page. "noindex" is a meta tag inside the HTML (or the X-Robots-Tag header) and acts after: the crawler downloads it, reads it, and decides not to index it. Use Disallow to save crawl budget on areas with no search value — catalogue filters, internal search results, admin panels. Use noindex for pages that must leave search but stay reachable, such as thank-you pages and print versions.
Do I need to declare the sitemap in robots.txt?
It is not required, but it is the cheapest way to guarantee that any crawler finds the full list of URLs. A single line — "Sitemap: https://example.com/sitemap.xml" — anywhere in the file is enough; it is global and belongs to no User-agent group. Without it you depend on registering the sitemap by hand in each webmaster tool, which covers Google and Bing and does nothing for anyone else.
Which rule wins when several groups match?
A crawler applies exactly one group: the most specific User-agent matching its name. If there is a "Googlebot" group and a "*" group, Googlebot follows only the first and ignores the second entirely — including rules that exist only in the wildcard. This is the source of a common silent mistake: someone adds "Disallow: /admin/" to the "*" group assuming it covers everyone, but Googlebot, which has its own group, keeps crawling right in. Within the chosen group, the longest matching path wins, and a tie between Allow and Disallow goes to Allow.
Does robots.txt protect confidential content?
No — and using it that way makes things worse. The file is public and readable by anyone, so listing "Disallow: /private-area/" publishes an index of exactly the paths you want hidden. Anyone probing for a way in reads robots.txt first for that very reason. Confidential content is protected with server-side authentication. robots.txt exists to guide crawling, not to control access.
## other tools