ubricHelp

How do I allow AI bots in robots.txt?

Rubric check: robots.txt that blocks a search or AI search bot (check id robots)

Don't disallow the bots that fetch pages for AI answers and search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and Claude-User (Claude), PerplexityBot, Googlebot and Bingbot. Rubric flags your robots.txt when it blocks any of them.

Free, no card. Your report opens with this check first.

Why it matters for SEO

robots.txt tells crawlers which paths they may fetch. Blocking Googlebot or Bingbot keeps your pages out of their results. A stray Disallow: / under User-agent: * blocks every bot at once, which often happens when a staging robots.txt goes live.

Does this matter for ChatGPT and AI Overviews?

Yes, directly. Each AI search tool uses its own bot to fetch pages, and a blocked bot means that tool can't read or cite you. Training bots such as GPTBot, ClaudeBot and Google-Extended are a separate choice: blocking them keeps your content out of model training, and you can still be cited through the search bots. Rubric only names those as a trade-off.

How Rubric scores it

GateA gate costs no points of its own. When it fails, it can cap the score.

Why this tier: An engine whose search bot is blocked can't read or cite your pages.

It is read from your robots.txt file.

When robots.txt blocks Googlebot or four or more AI engines, the whole site's score is capped at 25. A block on fewer engines lowers only those engines' scores.

From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).

How common it is

Found on 6 of 364 sites where Rubric could judge it (1.7%). That is 5 of 287 judged sites in Rubric's 313-site test set and 1 of 77 sites audited on Rubric between 11 Nov 2026 and 17 Nov 2026 under the current check rules (scoring 2026-11k or later). Measured 17 Nov 2026 with scoring 2026-v2.1.

1.7% of the sites Rubric could judge have this problem. Check yours.

Free, no card. Your report opens with this check first.

How accurate the check is

Lab test: passed. On test pages built with this problem, it caught 2 of 2. On clean test pages, it stayed quiet 2 of 2 times. (Engine build bbcec7c.)

Real sites: people checked 4 of its findings by hand and 1 was wrong (25%). Last checked 14 Nov 2026. That's a small sample, so treat the rate as rough.

Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.

How to fix it

WordPress

  1. Open yoursite.com/robots.txt and look for Disallow lines under the bots above or under User-agent: *.
  2. Edit it in Yoast SEO (Yoast SEO > Tools > File editor) or Rank Math (Rank Math SEO > General Settings > Edit robots.txt), or edit the robots.txt file on your server.
  3. Check Settings > Reading too: "Discourage search engines from indexing this site" should be unticked on a live site.

Shopify

  1. Shopify's default robots.txt allows these bots. If yours blocks one, the theme has a custom robots.txt.liquid.
  2. Go to Online Store > Themes, open the menu next to your theme, choose Edit code and open templates/robots.txt.liquid.
  3. Remove the rule blocking the bot and save. Deleting the template restores Shopify's default.

Webflow

  1. Go to Site settings > SEO > Indexing and find the robots.txt box.
  2. Remove any Disallow rule aimed at the bots above, or a Disallow: / under User-agent: *.
  3. Save and publish the site.

Custom code

Allow the search bots, and decide about training bots separately:

text
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

A bot follows only the most specific group that names it, so check each bot's group, not just User-agent: *.

More robots.txt checks · The full AI SEO glossary

Free, no card. Your report opens with this check first.