Free AI crawler check: can ChatGPT, Claude and Perplexity read your site?
Paste a page. In about ten seconds you’ll see which AI crawlers your robots.txt lets in, line by line, whether the page tells search engines not to index or quote it, and whether the content is in the HTML or hidden behind JavaScript. Free, no sign-up. Works for Shopify stores too.
What the results mean
- Blocked means a robots.txt rule shuts that crawler out of this page. We quote the rule and its line number.
- Limited means the crawler can get in but something gets in the way: nosnippet, content that needs JavaScript, or a server that refused our test request.
- Can’t tell means we couldn’t read robots.txt or the page. We won’t guess.
Search crawlers and training crawlers are different. Blocking a training crawler like GPTBot doesn’t take you out of ChatGPT search. Whether to allow training is your call, so the fix we suggest leaves those rules alone.
A pass means the crawlers can get in and the content is there to read. It doesn’t mean an engine will quote you. That depends on the page itself.
Questions
What does the AI crawler check look at?
Four things for the page you paste: what your robots.txt tells each AI crawler, with the exact line; how your server answers a request using each crawler's name; whether the page has noindex or nosnippet; and whether the main content is in the HTML or only appears after JavaScript runs.
If I block GPTBot, am I out of ChatGPT?
No. GPTBot collects text for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens a page when someone asks ChatGPT about it. Claude works the same way: ClaudeBot is for training, Claude-SearchBot and Claude-User are for answers. You can opt out of training and still be readable in search.
Does Google-Extended affect AI Overviews?
No. Google says Google-Extended controls whether your content is used to train Gemini models and for grounding in Gemini apps. It doesn't affect Google Search, and AI Overviews come from Google Search, which Googlebot crawls. Only noindex or nosnippet keep a page out of AI Overviews, and they affect normal results too.
Why is a firewall refusal a warning, not a block?
Our test requests come from our server, not from OpenAI's or Anthropic's. Many firewalls refuse anyone imitating a bot and still let the real crawler through after checking its IP address. So a refusal is a reason to check your firewall or CDN bot settings, not proof of a block. We only call something blocked when a robots.txt rule says so, and we quote the line.
I'm on Shopify. What should I look for?
Shopify's default robots.txt lets AI crawlers read products, collections, pages and blogs. Problems usually come from an edited robots.txt.liquid, a store that's still password-protected, or a theme or app adding a noindex tag. We don't send test requests with bot names to Shopify stores: Shopify filters fake bot traffic, so a refusal would prove nothing.
Why does JavaScript matter?
In Vercel’s analysis of AI crawler traffic (December 2024), the crawlers from OpenAI, Anthropic and Perplexity didn’t run JavaScript. If your content only appears after scripts run, those crawlers get a near-empty page. We count the words of main content in the raw HTML. We can’t run JavaScript here, so a short page with no sign of a JavaScript app is reported as “can’t tell”, not as a problem.
Is this the same as the Rubric audit?
No. This checks whether AI crawlers can get to one page. The free Rubric audit crawls your site and estimates how quotable each page is. It doesn't count citations and can't promise any.
What do you store?
The URL you checked and the result, so a repeat check within the hour is instant, plus a one-way hash of your IP address for rate limits. No account, no email, and not your IP address itself.