Cloudflare and OpenAI: Broader page discovery, more efficient search crawling experiments

1. An experiment that discovered hundreds of millions of new pages

The companies began a research pilot in July 2026 and released initial results on October 15 of that year. Cloudflare says the experiment covered a participating group representing about 1% of its entire network. OpenAI discovered hundreds of millions of webpages it had not previously known about and added them to its search index. An index is a list of pages collected for use in search.

Cloudflare provides signals that aid page discovery, while OpenAI independently assesses and ranks pages for relevance and usefulness. The network supplies candidate addresses, while OpenAI decides how to present them in search results. Source links in ChatGPT Search offer users a route to the original content and its site.

2. Looking beyond addresses to page status

An address does not guarantee a useful page. The original post cites bots repeatedly requesting nonexistent files, sites redirecting nearly all addresses to the homepage, and multiple addresses pointing to the same content. Identifying these patterns can reduce unnecessary visits before crawlers download content. A crawler is a program that automatically visits and collects webpages.

Change signals help determine whether a page is worth revisiting. For example, rather than repeatedly downloading unchanged pages, they help direct crawling resources toward pages that have changed.

Signals flagging potentially harmful pages are also being tested. If an attacker plants a fake login page on a legitimate site, this approach flags that page rather than the entire site. Initial results suggest these signals could help identify pages that should be excluded from the index.

3. The benefits of broader discovery and the crawling load

Discovering new pages may initially increase crawling as they are evaluated and indexed. Reducing repeated crawling of unchanged pages, however, can lower unnecessary site load and focus OpenAI's search resources on useful information. The experiment aims to achieve broader discovery with less waste, not simply to reduce crawling across the board.

Regardless of how many pages were discovered, no quantitative results were provided showing reductions in crawling volume or server load. The discovery of hundreds of millions of pages should therefore not be interpreted as direct evidence of lower operating costs.

4. Operators' choices and remaining validation

Site operators can use AI Crawl Control to monitor and manage AI systems' access to their sites. However, the original post provides no instructions for joining this pilot, specific setup procedures, or code examples. The announcement aims to improve discovery opportunities for operators who choose to make their content available to AI.

Having signals about a page does not grant permission to access it. This research uses web metadata for search and discovery rather than sharing page content for model training. Metadata is information about a page, distinct from the page's content itself.

Next steps include improving page selection, testing faster and more reliable change signals, and measuring outcomes after content is discovered. How broader discovery translates into actual business results remains a question for both companies to examine.

These initial results suggest potential for broader search sources, but their operational value depends on reducing unnecessary crawling load while respecting access permissions.