Is my firewall blocking Googlebot or AI crawlers?
Rubric check: page blocked or challenged for a search or AI bot (check id bot_blocked)
It might be. Rubric fetches your pages as a search or AI bot and as a browser. When the bot gets a block, a challenge page or a different page from the one people see, Rubric flags it.
Why it matters for SEO
robots.txt only asks bots to stay away. A firewall, CDN or security plugin can stop them at the door, whatever robots.txt says. If Googlebot or Bingbot gets a challenge page instead of your content, those pages can drop out of search. Blocking a bot on purpose is your choice, and you can ignore the finding.
Does this matter for ChatGPT and AI Overviews?
Yes, directly. If OAI-SearchBot, Claude-SearchBot or PerplexityBot gets a challenge page, the AI tool has nothing to read and cites someone else. Rubric can only send the bot's name, not its IP address, so a firewall that lets verified bots in by IP may still let the real bot through.
How Rubric scores it
MajorMajor checks cost twice as much as a standard check when they fail.
Why this tier: A bot that is blocked or challenged can't read the page, whatever robots.txt says.
It is judged on every page Rubric scores.
Every Rubric score starts at 100. When this check fails, it takes up to 24 points off. Even one failing page costs at least 12 points (half of the most), and the rest comes off as the share of affected pages grows. A warning costs less than a failure.
This check doesn't cap your score. It only takes off its own points.
From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).
How common it is
Rubric doesn't have enough crawled sites to say how common this is yet.
How accurate the check is
Lab test: passed. On test pages built with this problem, it caught 2 of 2. On clean test pages, it stayed quiet 2 of 2 times. (Engine build bbcec7c.)
Real sites: people checked 11 of its findings by hand and 7 were wrong (64%). Last checked 14 Nov 2026. That's a small sample, so treat the rate as rough.
Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.
How to fix it
WordPress
- Check your security plugin (Wordfence, Sucuri, All in One Security or similar) for bot-blocking or rate-limit rules and allow the bots you want.
- Check your host's firewall settings, and your CDN if you use one.
- On Cloudflare, look at Security > Bots and the AI crawler settings, and make sure the search and AI bots you want aren't blocked or challenged.
Shopify
- Shopify controls the storefront's firewall and has no setting for bot rules.
- Check your store isn't password protected: Online Store > Preferences > Password protection.
- If you put a CDN or proxy in front of Shopify, check its bot rules. If Rubric still reports a block, contact Shopify Support with the bot name and URL.
Webflow
- Webflow manages the firewall for sites it hosts. Standard plans have no setting for bot rules.
- Check the page isn't password protected in its page settings, and the site isn't behind a staging password.
- If you route the domain through your own Cloudflare or another proxy, check its bot rules. Otherwise contact Webflow Support with the bot name and URL.
Custom code
Test the page as a bot yourself:
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/A 403, 429 or a challenge page means a rule is catching the bot. Allow the bots you want in your firewall or CDN, ideally by their published IP ranges or verified-bot lists, not by name alone.