Is there a size limit for robots.txt?
Rubric check: robots.txt over the size limit (check id ne_robots_size_limit)
Yes. Google reads only the first 500 KiB of a robots.txt file and ignores everything after that. If yours is bigger, the rules near the end don't apply to Googlebot at all.
Why it matters for SEO
A rule that Google never reads is the same as no rule. If a Disallow for your admin or search pages sits past the limit, Google may crawl them. If an Allow sits there, a page you wanted crawled may stay blocked by an earlier rule. Files this big are almost always generated, with one line per URL.
Does this matter for ChatGPT and AI Overviews?
Other crawlers, including AI ones, don't all publish a size limit, so you can't count on them reading a huge file either. A short file with pattern rules behaves the same for every bot.
How Rubric scores it
StandardStandard checks are the baseline every other tier is measured against.
Why this tier: robots.txt over 500 KB is cut off.
It is read from your robots.txt file.
Every Rubric score starts at 100. When this check fails, it takes up to 12 points off. A warning takes off less than a failure.
This check doesn't cap your score. It only takes off its own points.
From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).
How common it is
Found on 0 of 288 sites where Rubric could judge it (0%). That is 0 of 275 judged sites in Rubric's 313-site test set and 0 of 13 sites audited on Rubric between 11 Nov 2026 and 17 Nov 2026 under the current check rules (scoring 2026-11k or later). Measured 17 Nov 2026 with scoring 2026-v2.1.
How accurate the check is
Lab test: passed. On test pages built with this problem, it caught 2 of 2. On clean test pages, it stayed quiet 1 of 1 times. (Engine build bbcec7c.)
Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.
How to fix it
WordPress
- Open your robots.txt. In Yoast SEO it's under Yoast SEO > Tools > File editor; in Rank Math under Rank Math > General Settings > Edit robots.txt. If neither, it's a file in your site's root folder.
- Replace long lists of single URLs with wildcard rules, for example
Disallow: /*?filter=. - Check a security or SEO plugin isn't adding lines automatically.
Shopify
- Shopify's default robots.txt is small. A large file means someone edited the
robots.txt.liquidtemplate. - Open Online Store > Themes > Edit code, find
robots.txt.liquidunder Templates and remove the long lists of single URLs. Use pattern rules instead. - If you're not sure what's custom, delete the template to go back to Shopify's default.
Webflow
- Go to Site settings > SEO > Indexing and open the robots.txt field.
- Replace long URL lists with wildcard rules, then save and publish.
Custom code
Group rules with * (any characters) and $ (end of URL):
User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*.pdf$
Sitemap: https://example.com/sitemap.xmlKeep the most important rules at the top. Check the size with curl -s https://example.com/robots.txt | wc -c.