ubricHelp

Is there a size limit for robots.txt?

Rubric check: robots.txt over the size limit (check id ne_robots_size_limit)

Yes. Google reads only the first 500 KiB of a robots.txt file and ignores everything after that. If yours is bigger, the rules near the end don't apply to Googlebot at all.

Free, no card. Your report opens with this check first.

Why it matters for SEO

A rule that Google never reads is the same as no rule. If a Disallow for your admin or search pages sits past the limit, Google may crawl them. If an Allow sits there, a page you wanted crawled may stay blocked by an earlier rule. Files this big are almost always generated, with one line per URL.

Does this matter for ChatGPT and AI Overviews?

Other crawlers, including AI ones, don't all publish a size limit, so you can't count on them reading a huge file either. A short file with pattern rules behaves the same for every bot.

How Rubric scores it

StandardStandard checks are the baseline every other tier is measured against.

Why this tier: robots.txt over 500 KB is cut off.

It is read from your robots.txt file.

Every Rubric score starts at 100. When this check fails, it takes up to 12 points off. A warning takes off less than a failure.

This check doesn't cap your score. It only takes off its own points.

From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).

How common it is

Found on 0 of 288 sites where Rubric could judge it (0%). That is 0 of 275 judged sites in Rubric's 313-site test set and 0 of 13 sites audited on Rubric between 11 Nov 2026 and 17 Nov 2026 under the current check rules (scoring 2026-11k or later). Measured 17 Nov 2026 with scoring 2026-v2.1.

0% of the sites Rubric could judge have this problem. Check yours.

Free, no card. Your report opens with this check first.

How accurate the check is

Lab test: passed. On test pages built with this problem, it caught 2 of 2. On clean test pages, it stayed quiet 1 of 1 times. (Engine build bbcec7c.)

Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.

How to fix it

WordPress

  1. Open your robots.txt. In Yoast SEO it's under Yoast SEO > Tools > File editor; in Rank Math under Rank Math > General Settings > Edit robots.txt. If neither, it's a file in your site's root folder.
  2. Replace long lists of single URLs with wildcard rules, for example Disallow: /*?filter=.
  3. Check a security or SEO plugin isn't adding lines automatically.

Shopify

  1. Shopify's default robots.txt is small. A large file means someone edited the robots.txt.liquid template.
  2. Open Online Store > Themes > Edit code, find robots.txt.liquid under Templates and remove the long lists of single URLs. Use pattern rules instead.
  3. If you're not sure what's custom, delete the template to go back to Shopify's default.

Webflow

  1. Go to Site settings > SEO > Indexing and open the robots.txt field.
  2. Replace long URL lists with wildcard rules, then save and publish.

Custom code

Group rules with * (any characters) and $ (end of URL):

text
User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*.pdf$
Sitemap: https://example.com/sitemap.xml

Keep the most important rules at the top. Check the size with curl -s https://example.com/robots.txt | wc -c.

More robots.txt checks · The full AI SEO glossary

Free, no card. Your report opens with this check first.