Let Rubric through your firewall

Some firewalls block every crawler they don't recognise. That's often the right call for an online pharmacy, a bank or a busy store. It also means Rubric sees a bot check instead of your pages, so it can't score them.

Verified crawling fixes the audit without opening your site to anyone else. Your firewall lets through requests that carry a secret header only your Rubric account has. Everything else, real AI bots included, still meets your normal rules.

How it works

Rubric sends the header only to the domains on your list and their subdomains. If a page redirects to another host, the header is removed before the redirect is followed. If you audit someone else's site, they never see your token.

What Rubric sends on every verified request:

X-Rubric-Verify: YOUR_RUBRIC_TOKEN
User-Agent: Mozilla/5.0 (compatible; RubricVerified/1.0; +https://rubric.gogochimp.com/docs/firewall)

Match on the header, not the user agent. Anyone can copy a user agent. Only your account has the token.

Why not an IP allow-list?

Rubric's crawler doesn't run from a fixed IP address today, and we don't publish IP ranges, so an IP rule would break without warning. Use the header. If your firewall can only allow by IP, email us and tell us which product you use.

Your report still shows the AI-bot problem

The header only lets Rubric through. GPTBot, ClaudeBot, PerplexityBot, Google and Bing don't carry it. Rubric tests those bots separately with their own user agents and no token, so if your firewall blocks them, the report still says so. For a pharmacy or a store, that finding is usually the point: decide which AI bots you want to let in, then allow them by their published verification methods.

Cloudflare

(any(http.request.headers["x-rubric-verify"][*] eq "YOUR_RUBRIC_TOKEN"))

Cloudflare's free-plan Bot Fight Mode can't be skipped by a rule. If it's on, Rubric (and every other verified crawler) stays blocked: switch it off under Security → Bots, or use Super Bot Fight Mode on a paid plan, which a Skip rule can bypass. Header names in Cloudflare expressions are lower case.

AWS WAF

Add a rule to your web ACL with a lower priority number than any Bot Control or rate-based rule, so it's evaluated first:

{
  "Name": "rubric-verified-crawler",
  "Priority": 0,
  "Action": {
    "Allow": {}
  },
  "Statement": {
    "ByteMatchStatement": {
      "FieldToMatch": {
        "SingleHeader": {
          "Name": "x-rubric-verify"
        }
      },
      "PositionalConstraint": "EXACTLY",
      "SearchString": "YOUR_RUBRIC_TOKEN",
      "TextTransformations": [
        {
          "Priority": 0,
          "Type": "NONE"
        }
      ]
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "rubric-verified-crawler"
  }
}

In the console: Rule builder → If a request matches the statement → Inspect Single header x-rubric-verify → Match type Exactly matches string → your token → Action Allow.

Akamai

Menu names differ between App & API Protector, Kona Site Defender and Bot Manager versions, so treat this as the shape of the rule and check with whoever runs your Akamai config:

• Bot Manager: add a custom bot (or custom client) that matches request header X-Rubric-Verify equals your token, and set its action to Allow.
• App & API Protector / WAF rules: add an exception (or a match target bypass) for the same header condition, so the rate controls and attack groups don't block it.
• Activate the change on staging first, re-run the Rubric audit against staging if you have one, then promote to production.

Imperva (Cloud WAF)

Fastly

Next-Gen WAF: Rules → Site rules → Add rule → type Request. Condition: Request header X-Rubric-Verify equals your token. Action: Allow request. Put it above your bot and rate-limit rules.

VCL (Fastly CDN with your own bot logic): skip your bot checks when the header matches, at the top of vcl_recv:

if (req.http.X-Rubric-Verify == "YOUR_RUBRIC_TOKEN") {
  set req.http.X-Rubric-Verified = "1";
}
# then wrap your bot / challenge logic:
# if (!req.http.X-Rubric-Verified) { ...your existing checks... }

nginx

Map the header to a flag, then let flagged requests past your bot block and rate limit:

# http { } block
map $http_x_rubric_verify $rubric_verified {
    default        0;
    "YOUR_RUBRIC_TOKEN"  1;
}
# rate limiting: an empty key is never limited
map $rubric_verified $limit_key {
    1 "";
    0 $binary_remote_addr;
}
limit_req_zone $limit_key zone=perip:10m rate=10r/s;

# server { } block: your existing user-agent block, with the exception
set $block_bot 0;
if ($http_user_agent ~* "(bot|crawler|spider)") { set $block_bot 1; }
if ($rubric_verified) { set $block_bot 0; }
if ($block_bot) { return 403; }

Apache

With ModSecurity, turn the rule engine off for matching requests (pick an unused rule id):

SecRule REQUEST_HEADERS:X-Rubric-Verify "@streq YOUR_RUBRIC_TOKEN" \
    "id:1009001,phase:1,pass,nolog,ctl:ruleEngine=Off"

Without ModSecurity, exempt it from your own bot rule:

<If "%{HTTP:X-Rubric-Verify} == 'YOUR_RUBRIC_TOKEN'">
    Require all granted
</If>
<Else>
    # your existing bot / user-agent rules
</Else>

Other reverse proxies and CDNs

The rule is always the same: if the request header X-Rubric-Verify exactly equals your token, skip bot protection, challenges and rate limits for that request. Put the rule before your bot rules. Don't log the header value.

Security

• One token per account. Only your Rubric account has it. It's never put in a report, an export or a share link.
• Revocable. Revoke it or make a new one in Account → Firewall. Rubric reads it fresh for every crawl, so the next crawl stops sending the old one. Remove or update the firewall rule at the same time.
• Only your domains. It's sent only to the domains you list and their subdomains, never to a site a page links or redirects to.
• Read-only. The crawler only makes GET and HEAD requests. It never submits forms, logs in or posts anything.
• Gentle. A verified crawl keeps at most 2 requests open at a time, backs off when your server sends 429 or 503 (it honours Retry-After), and stops at your plan's page cap.
• robots.txt. Rubric reads your robots.txt and leaves pages it disallows for all crawlers out of your score (they're listed separately). It still fetches them, so it can tell you what's behind the rule.
• Pro feature. Free accounts get the firewall finding and this guide. Creating a token needs Rubric Pro; if Pro ends, Rubric stops sending the token.

Check it worked

Re-run the audit. A verified report has a Verified crawler line at the top. If the audit still says "Not scored: bot check" and adds "This crawl sent your Rubric verification header and was still blocked", the rule isn't matching yet: check the header name (X-Rubric-Verify, lower case in Cloudflare), the exact token, that the rule sits above your bot rules, and that it covers both the bare domain and www. In your server logs, verified requests show the user agent RubricVerified/1.0.