Cloudflare and OpenAI Pilot Improves AI Web Discovery and Site Owner Control

What changed?

Cloudflare and OpenAI launched a joint pilot involving roughly 1% of Cloudflare's network. The collaboration enables OpenAI's search systems to discover and index hundreds of millions of new web pages, specifically for use within ChatGPT. Cloudflare contributes network-level signals that make it easier for OpenAI crawlers to find relevant and helpful pages. The pilot also tests methods of flagging potentially harmful web pages to improve safety. Website owners now have improved management of AI access through Cloudflare's AI Crawl Control. The data used is limited to metadata for indexing and search, and does not permit full content access or model training.

Why does it matter to an everyday developer?

This update directly affects how developer-created content can be discovered and surfaced in ChatGPT and similar AI search tools. Improved discovery means reference guides, product docs, or app documentation hosted on a developer's website have higher chances of being found and cited by ChatGPT. Enhanced signals reduce the risk of unnecessary crawling, minimizing server load from bots. The addition of safety signals helps prevent compromised or malicious pages from being surfaced. Developers and site owners gain finer control over how and whether AI systems interact with their websites through Cloudflare's tools.

What can the developer do now?

Developers who host their websites with Cloudflare can use the AI Crawl Control feature to monitor and adjust how AI systems attempt to access their content. By configuring crawl rules, developers can enable or restrict AI discovery based on their preferences. It is also advisable to review site structure and metadata to maximize discoverability and ensure helpful, searchable web content. Additionally, developers should monitor Cloudflare announcements for expanded network pilot participation or evolving AI access features.