ubricHelp

Robots.txt

Your robots.txt file tells each crawler which addresses it may fetch. These checks cover the rules for the bots that fetch pages for AI answers and search, the size limit Google reads up to, and the crawl-delay rule Google ignores but Bing follows.
  • How do I allow AI bots in robots.txt?
    Don't disallow the bots that fetch pages for AI answers and search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and Claude-User (Claude), PerplexityBot, Googlebot and Bingbot. Rubric flags your robots.txt when it blocks any of them.
    Gate · check id robots
  • Is there a size limit for robots.txt?
    Yes. Google reads only the first 500 KiB of a robots.txt file and ignores everything after that. If yours is bigger, the rules near the end don't apply to Googlebot at all.
    Standard · check id ne_robots_size_limit
  • Does Google respect crawl-delay in robots.txt?
    No. Google ignores Crawl-delay in robots.txt. Bing obeys it, so Bingbot fetches your pages no faster than the delay allows. Rubric flags it as a note when the rule applies to Googlebot, Bingbot or all bots.
    Minor · check id ne_robots_crawl_delay_note

All Rubric checks in the AI SEO glossary