ubricHelp

Is my firewall blocking Googlebot or AI crawlers?

Rubric check: page blocked or challenged for a search or AI bot (check id bot_blocked)

It might be. Rubric fetches your pages as a search or AI bot and as a browser. When the bot gets a block, a challenge page or a different page from the one people see, Rubric flags it.

Free, no card. Your report opens with this check first.

Why it matters for SEO

robots.txt only asks bots to stay away. A firewall, CDN or security plugin can stop them at the door, whatever robots.txt says. If Googlebot or Bingbot gets a challenge page instead of your content, those pages can drop out of search. Blocking a bot on purpose is your choice, and you can ignore the finding.

Does this matter for ChatGPT and AI Overviews?

Yes, directly. If OAI-SearchBot, Claude-SearchBot or PerplexityBot gets a challenge page, the AI tool has nothing to read and cites someone else. Rubric can only send the bot's name, not its IP address, so a firewall that lets verified bots in by IP may still let the real bot through.

How Rubric scores it

MajorMajor checks cost twice as much as a standard check when they fail.

Why this tier: A bot that is blocked or challenged can't read the page, whatever robots.txt says.

It is judged on every page Rubric scores.

Every Rubric score starts at 100. When this check fails, it takes up to 24 points off. Even one failing page costs at least 12 points (half of the most), and the rest comes off as the share of affected pages grows. A warning costs less than a failure.

This check doesn't cap your score. It only takes off its own points.

From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).

How common it is

Rubric doesn't have enough crawled sites to say how common this is yet.

Check yours.

Free, no card. Your report opens with this check first.

How accurate the check is

Lab test: passed. On test pages built with this problem, it caught 2 of 2. On clean test pages, it stayed quiet 2 of 2 times. (Engine build bbcec7c.)

Real sites: people checked 11 of its findings by hand and 7 were wrong (64%). Last checked 14 Nov 2026. That's a small sample, so treat the rate as rough.

Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.

How to fix it

WordPress

  1. Check your security plugin (Wordfence, Sucuri, All in One Security or similar) for bot-blocking or rate-limit rules and allow the bots you want.
  2. Check your host's firewall settings, and your CDN if you use one.
  3. On Cloudflare, look at Security > Bots and the AI crawler settings, and make sure the search and AI bots you want aren't blocked or challenged.

Shopify

  1. Shopify controls the storefront's firewall and has no setting for bot rules.
  2. Check your store isn't password protected: Online Store > Preferences > Password protection.
  3. If you put a CDN or proxy in front of Shopify, check its bot rules. If Rubric still reports a block, contact Shopify Support with the bot name and URL.

Webflow

  1. Webflow manages the firewall for sites it hosts. Standard plans have no setting for bot rules.
  2. Check the page isn't password protected in its page settings, and the site isn't behind a staging password.
  3. If you route the domain through your own Cloudflare or another proxy, check its bot rules. Otherwise contact Webflow Support with the bot name and URL.

Custom code

Test the page as a bot yourself:

bash
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/

A 403, 429 or a challenge page means a rule is catching the bot. Allow the bots you want in your firewall or CDN, ideally by their published IP ranges or verified-bot lists, not by name alone.

More bot access checks · The full AI SEO glossary

Free, no card. Your report opens with this check first.