Free tool ยท No sign-up
Robots.txt and indexability checker
Can Google and AI crawlers reach this page? Test robots.txt the way Google reads it, check noindex, canonical and status codes, and see which of 18 crawlers your firewall or CDN turns away.
Result
See every crawler and the fixes
Confirm your email to see which crawlers are allowed or blocked, by which rule, and what to change.
- Googlebot, Bingbot and 16 AI crawlers: robots.txt rule and live answer for each
- The exact robots.txt line that blocks a crawler
- A fix list in order of impact
- A copy by email to send to your developer
Full result unlocked for . This browser is remembered, so next time it opens straight away.
What to fix or review
Crawler by crawler
| Crawler | Used for | robots.txt | Live request |
|---|
Page details
Why a page drops out of Google
Google retired its own robots.txt Tester in 2023, so most people now find a blocking rule only after traffic falls. The usual causes are small: a Disallow line left over from a staging site, a noindex tag ticked by mistake in an SEO plugin, a canonical tag pointing at another page, or a firewall that treats crawlers as attackers.
This checker looks at each of these for one address:
- Status code: only pages that answer 200 are indexed. Redirects are followed and reported.
- robots.txt: read with Google's own matching rules for Googlebot, Bingbot and 16 AI crawlers.
- noindex: in the robots and googlebot meta tags and in the X-Robots-Tag header.
- Canonical: whether it points to the page itself, to another live page or to a broken one.
- Live crawler test: a request with each crawler's user agent, to catch CDN and firewall blocks that robots.txt does not show.
Pages that pass but still get no traffic usually need better titles and content: run the SEO checker next, and preview your result in Google with the search preview tool.
Questions
Does this tell me whether my page is in Google?
No, and no outside tool can say for sure: only Google Search Console knows, and Google does not allow automated searches to check. This tool answers the question before that one: can Google crawl and index the page at all. If every line is green and the page is still missing after a few weeks, use URL Inspection in Search Console.
How does Google read robots.txt?
Googlebot follows one group of rules: the one whose user-agent line matches it most specifically, or the "*" group if none does. Within that group, the rule with the longest matching path wins, and Allow wins a tie. "*" matches anything and "$" marks the end of the address. The tester above uses the same rules.
Is a page blocked in robots.txt kept out of Google?
Not always. robots.txt stops crawling, not indexing: a blocked page that other sites link to can still appear in results, usually without a description. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header.
Why does an AI crawler show as blocked when my robots.txt allows it?
Because the block happens before robots.txt is read. Since July 2025 Cloudflare blocks known AI crawlers by default on new sites, and firewalls and security plugins often do the same. Our live test sends a request with each crawler's user agent; a 403 or 503 where a browser gets 200 points to that kind of block.
Should I block AI crawlers?
It depends on what you want. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect content for AI models. Search and answer crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot fetch pages to answer questions and can send you visitors. Many businesses block training and allow answer crawlers.
Could the live test be wrong?
Yes, in one direction. Some firewalls check that a request claiming to be Googlebot really comes from Google, and turn our test away even though the real Googlebot gets through. Treat a blocked result as a reason to check your firewall settings and Search Console, not as proof.
Pages missing from Google?
Our technical SEO team finds why pages are not indexed, fixes crawl and canonical problems, and sets up AI crawler rules that match your goals.