Does Google respect crawl-delay in robots.txt?
Rubric check: crawl-delay in robots.txt (check id ne_robots_crawl_delay_note)
No. Google ignores Crawl-delay in robots.txt. Bing obeys it, so Bingbot fetches your pages no faster than the delay allows. Rubric flags it as a note when the rule applies to Googlebot, Bingbot or all bots.
Why it matters for SEO
For Google, the line does nothing. Google sets its own crawl rate from how fast your server responds. For Bing, a delay of several seconds per page means a large site can take a long time to be crawled in full.
That's fine if your server truly needs it. If it doesn't, the delay just slows how fast Bing sees new and updated pages.
Does this matter for ChatGPT and AI Overviews?
This one can matter for Copilot. Copilot answers from Bing's index, so a long crawl-delay can slow how fast your updates reach it.
Google's AI Overviews don't depend on Bingbot's pace, because Google ignores the rule. Rubric shows this as a minor note, not an error.
How Rubric scores it
MinorMinor checks cost a quarter to a half of a standard check when they fail.
Why this tier: Google ignores crawl-delay, but Bing obeys it, so a long delay slows Bing's view of new pages.
It is read from your robots.txt file, for Googlebot, Bingbot and the * group.
Every Rubric score starts at 100. When this check fails, it takes up to 3 points off. A warning takes off less than a failure.
All the minor checks share one limit: together they can take off at most 25 points, however many of them fail.
This check doesn't cap your score. It only takes off its own points.
From the engine's scoring files, scoring 2026-v2.1 (engine d5c591d).
How common it is
Found on 16 of 288 sites where Rubric could judge it (5.6%). That is 16 of 275 judged sites in Rubric's 313-site test set and 0 of 13 sites audited on Rubric between 11 Nov 2026 and 17 Nov 2026 under the current check rules (scoring 2026-11k or later). Measured 17 Nov 2026 with scoring 2026-v2.1.
How accurate the check is
Lab test: passed. On test pages built with this problem, it caught 3 of 3. On clean test pages, it stayed quiet 2 of 2 times. (Engine build bbcec7c.)
Holdout test 5 (sites the check had never seen): 0 of 9 findings judged wrong.
Lab as of 16 Nov 2026; judged findings as of 15 Nov 2026.
How to fix it
WordPress
- Open
https://yoursite.com/robots.txtto see the rule. - If your SEO plugin manages robots.txt (Yoast SEO: Yoast SEO > Tools > File editor; Rank Math: Rank Math > General Settings > Edit robots.txt), remove or lower the
Crawl-delayline there. - If there's a physical
robots.txtfile in your site's root folder, edit it with your host's file manager or SFTP.
Shopify
- Shopify's default robots.txt sets crawl-delay only for a few non-search bots, which Rubric doesn't flag.
- If Rubric reports it, your theme has a custom
robots.txt.liquid. Go to Online Store > Themes > Edit code, open Templates > robots.txt.liquid and remove theCrawl-delayline for Bingbot or*. - Save. Shopify serves the new file straight away.
Webflow
- Go to Site settings > SEO > Indexing.
- In the robots.txt box, remove or lower the
Crawl-delayline. - Save and publish.
Custom code
Remove the line, or keep a short delay only for the bot that strains your server:
User-agent: *
Disallow:
User-agent: Bingbot
Crawl-delay: 1To slow Bing without robots.txt, use Crawl Control in Bing Webmaster Tools, which lets you choose quieter hours.