Bot access
Before an engine can quote a page, its crawler has to be able to fetch it. These checks look for firewalls and bot protection that turn search or AI crawlers away, and for the Bing controls that decide whether Copilot may use your content.
- How do I stop or allow Copilot using my content?With Bing's robots rules.
ai-grounding:disallowin a Bingbot meta tag orX-Robots-Tagheader keeps a page out of Copilot answers. Until 1 Mar 2027 Bing also readsnoarchivethat way, andnocachelimits Copilot to the URL, title and snippet. Rubric flags any of these. - Is my firewall blocking Googlebot or AI crawlers?It might be. Rubric fetches your pages as a search or AI bot and as a browser. When the bot gets a block, a challenge page or a different page from the one people see, Rubric flags it.
- Does noarchive stop Copilot using my page?Yes, for now. Bing's Webmaster Guidelines say NOARCHIVE keeps a page out of Copilot answers, and NOCACHE lets Copilot use only its URL, title and snippet. Rubric flags pages that send either one.