Robots.txt
Your robots.txt file tells each crawler which addresses it may fetch. These checks cover the rules for the bots that fetch pages for AI answers and search, the size limit Google reads up to, and the crawl-delay rule Google ignores but Bing follows.
- How do I allow AI bots in robots.txt?Don't disallow the bots that fetch pages for AI answers and search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and Claude-User (Claude), PerplexityBot, Googlebot and Bingbot. Rubric flags your robots.txt when it blocks any of them.
- Is there a size limit for robots.txt?Yes. Google reads only the first 500 KiB of a robots.txt file and ignores everything after that. If yours is bigger, the rules near the end don't apply to Googlebot at all.
- Does Google respect crawl-delay in robots.txt?No. Google ignores
Crawl-delayin robots.txt. Bing obeys it, so Bingbot fetches your pages no faster than the delay allows. Rubric flags it as a note when the rule applies to Googlebot, Bingbot or all bots.