Cloudflare Blocks AI Crawlers by Default from 15 September. What Indian Brands Should Do Now

What this blog covers

On September 15, 2026, Cloudflare changed its default AI-crawler policy: training and agent crawlers are now blocked by default on ad-supported pages for new sign-ups, newly added sites, and free-plan customers who had not manually configured a preference, while search bots remain allowed. Paid customers with existing configurations were exempt from the automatic change. This piece explains what actually shifted, who it affects, and how Indian brands running on Cloudflare should check and set their own posture now.

Quick Answer: As of September 15, 2026, Cloudflare’s new default blocks AI training and agent crawlers on ad-supported pages for new customer sign-ups, newly added sites, and existing free-plan users who had not set a preference before the deadline; AI search bots remain allowed under the new default. Paid customers who had already configured their own settings were not affected by the automatic change. Any brand on Cloudflare, regardless of plan, should log into the dashboard now and explicitly confirm their AI Crawl Control settings rather than assume the default matches their intent.

What Actually Changed on September 15

Cloudflare shifted its default handling of AI crawler traffic for pages carrying ads: training and agent-type crawlers are now blocked by default rather than allowed by default, reversing the previous open-by-default posture. This follows the broader direction Cloudflare signaled in 2025 with its “Content Independence Day” announcement, and September 15, 2026 marks the date the default itself flipped for the accounts it applies to.

Search-oriented AI crawlers were treated differently and remained allowed under the new default, reflecting Cloudflare’s stated view that AI-powered search and AI training serve genuinely different purposes for a site owner.

Who Was Affected, and Who Was Exempt

The new default applied automatically to all new customer sign-ups after the change, any new site added by an existing customer, and existing free-plan customers who had not manually configured their own AI crawler preferences before the deadline. Paid customers who had already set explicit preferences, whether allowing or blocking, kept their own configuration and were not overridden.

This matters for Indian brands specifically because a meaningful share run marketing sites, blogs, and e-commerce storefronts on Cloudflare’s free or lower-tier plans, exactly the segment most likely to have been quietly switched to the new default without anyone on the marketing or SEO team noticing.

Search Bots vs. Training and Agent Crawlers

Cloudflare’s own framing draws a distinction that lines up closely with the broader industry debate on whether brands should block AI crawlers: search bots (the crawlers that power live citations in AI search results and answer engines) remained allowed by default, while training and agent crawlers (which ingest content for model improvement or autonomous browsing tasks, without a direct citation benefit) are the ones newly blocked by default on ad-supported pages.

For most commercial brands, this distinction is the one that actually matters when deciding whether to leave the new default in place or override it: losing training-crawler access carries limited, uncertain cost, while losing search-bot access carries a real, visible one, particularly since different AI engines already draw on different source pools, as why ChatGPT, Gemini, and AI Overviews almost never cite the same source shows, so losing access to even one engine’s crawler is not offset by staying visible to the others.

The “Pay Per Use” Piece, and Why It’s Still Early

Alongside the blocking-by-default shift, Cloudflare has been testing a “Pay Per Use” model with select partners, under which publishers could potentially earn compensation based on how their content is actually used by AI systems, rather than by simple crawl frequency. As of now, the mechanics, pricing, and eligibility remain experimental and not broadly available.

Brands should treat this as a development worth watching rather than something to plan around today. Blocking AI training crawlers now in anticipation of future compensation is a bet on a mechanism that does not yet exist in a mature, accessible form.

What Indian Brands Should Check This Week

Log into the Cloudflare dashboard and check the AI Crawl Control settings directly, rather than assuming the default matches your intent, whether you are on a free plan that may have shifted automatically or a paid plan where you may simply never have set a preference at all. Confirm the setting separately for search bots and for training/agent crawlers, since they are configured independently, and cross-check named crawlers like OpenAI’s GPTBot against the dashboard’s own crawler list rather than assuming every bot is categorised the way you’d expect.

If your brand’s growth depends on AI search visibility (most D2C and B2B brands, per the broader framework in allow, block, or charge), explicitly allow search bots rather than relying on the default holding. If you are a content-heavy publisher weighing the training-crawler tradeoff, this is a reasonable moment to make that decision deliberately rather than by default.

Getting This Wrong in Either Direction

Assuming the old open-by-default posture still applies, when it may have quietly flipped, risks losing AI search visibility without ever making an active decision to do so. Assuming the new block-by-default posture is automatically right for your brand risks the opposite: losing citation and shortlist visibility in a channel that, for most commercial brands, is worth more than the uncertain protection blocking provides.

Either mistake is avoidable with five minutes in the dashboard. The point of this update is not that Cloudflare made the wrong call; it is that any change to a shared infrastructure default is worth actively checking against your own business model rather than inheriting silently.

Search bots (AI Overviews, Perplexity, etc.): New default is Allowed. Recommended check: Confirm explicitly allowed if AI visibility matters to you.

Training crawlers: New default is Blocked on ad-supported pages. Recommended check: Decide deliberately based on content monetization model.

Agent crawlers: New default is Blocked on ad-supported pages. Recommended check: Review if you rely on AI agents completing tasks on your site.

Paid, previously configured accounts: New default is Unchanged (exempt from auto-flip). Recommended check: Still worth re-confirming settings are current.

Framework Explained

Check, don’t assume: infrastructure defaults change; the only reliable way to know your posture is to check the dashboard directly.

Separate search from training: these serve different purposes, and Cloudflare’s own default now treats them differently, so your review should too.

Don’t plan around Pay Per Use yet: it remains experimental; treat it as a future development, not a current strategy input.

Revisit after any platform-level change: this is unlikely to be the last default shift from Cloudflare or other infrastructure providers as the AI-crawler landscape matures.

GEO Readiness Note

This is a platform-policy explainer rather than a client-specific case study, since the change is recent enough that measured before/after visibility impact data is not yet available. Lyxel&Flamingo’s crawler-access recommendations, covered in allow, block, or charge, apply directly to how brands should respond to this shift.

Key Takeaways

On September 15, 2026, Cloudflare began blocking AI training and agent crawlers by default on ad-supported pages for new sign-ups, new sites, and unconfigured free-plan accounts.

AI search bots remained allowed under the new default; the distinction between search and training/agent crawlers is central to the change.

Paid customers with existing configurations were exempt from the automatic flip, yet should still re-confirm their settings.

Cloudflare’s “Pay Per Use” compensation model remains experimental and should not yet be a factor in crawler-access decisions.

Every brand on Cloudflare should actively check their AI Crawl Control dashboard rather than assume their prior posture still holds.

CXO Takeaway

If your site runs on Cloudflare, do you actually know whether your AI crawler settings changed underneath you on September 15? This is a five-minute dashboard check with real visibility consequences either way. Lyxel&Flamingo’s SEO team can review your current crawler configuration against your business model and confirm you are set up deliberately, not by default.