AI Crawler Controls Need Separate Policies for Search and Training

Summary

Cloudflare’s crawler controls now make the choice between blocking access and expressing a training preference more explicit. The distinction matters for search discoverability, and support still differs by operator: Bing’s robots.txt no-training support is planned for early 2027.

Key Updates

• Blocking now includes mixed-use crawlers. Cloudflare’s September 15 update makes Block and Block on pages with ads apply to crawlers such as Applebot, Bingbot and Googlebot, which serve both search and training purposes. Blocking them can therefore affect search discovery. Cloudflare says the controls are available to all customers on all plans.
• A training preference can preserve search access. Disallow AI Training uses Bot Preference Sync to publish applicable no-training preferences in robots.txt. Cloudflare says designated mixed-use crawlers remain allowed for search, while other training crawlers are blocked. Existing granular Training selections of Block or Block on pages with ads migrate to Disallow AI Training to preserve their practical effect; the migration does not mean those existing settings suddenly block search crawlers.
• Operator support is uneven. Cloudflare’s Accountable designation includes both existing capabilities and time-bound commitments. Apple and Google already provide robots.txt training opt-outs. Microsoft is building site-level support for Bing, targeted for early 2027. Until then, selecting Disallow AI Training does not automatically convey a no-training preference to Bing through robots.txt; Bing’s existing operator-specific controls are separate. Unified controls for AI summaries also remain planned.

Why It Matters

A crawler setting can answer two different questions: may this bot reach the site, and how may its operator use the content afterward? Blocking access can affect search indexing. Allowing access while publishing a training preference depends on the operator supporting and honoring that preference.

The operating assumption to revisit is that an “AI training” setting has one consistent effect across crawlers. This update gives site owners more explicit choices, but its label is not enough to establish the result. Cloudflare’s announcement describes implementation and commitments; it does not independently demonstrate downstream training compliance or guarantee search outcomes.

Builder Takeaway

Audit one domain’s effective crawler policy.

Write down the intended behavior for Search, Training and Agent traffic, then compare it with the migrated Cloudflare settings and published robots.txt rules. Check the relevant operator’s current controls, recording any gap—especially Bing’s pending robots.txt support. Review crawl activity and search reporting for unintended access changes. Use that check to identify mismatches before changing settings; successful crawling alone cannot prove that content was excluded from training.