Cloudflare signals helped OpenAI index hundreds of millions of new pages

A research pilot between Cloudflare and OpenAI helped OpenAI’s search crawler discover and index hundreds of millions of web pages it had not found before. The pilot drew on a cohort of roughly 1% of Cloudflare’s network. The companies have not said how often those pages were cited in ChatGPT or how much traffic they sent.

What did the Cloudflare and OpenAI pilot find?

The pilot showed that Cloudflare’s network signals can surface useful pages that ordinary link-based discovery misses, according to an October 15, 2026 post on Cloudflare’s blog coauthored by two OpenAI staff, as reported by Search Engine Journal. Cloudflare first announced the pilot on July 8.

• Discovery: signals such as how recently a page changed, what actually changed and the quality of a site’s traffic come from participating websites. Examples given include a washing machine manual, passport fee information and a restaurant’s menu and hours: pages with few inbound links.
• Crawl efficiency: change signals tell the crawler whether a revisit is worthwhile, aiming to cut requests for missing files, catch-all redirects and duplicate URLs. No crawl volume or server load figures were shared.
• Safety: the companies are also testing page-level signals to flag harmful content, such as a planted fake login form, without penalizing the whole domain.

The post is explicit that access rules still apply: “A signal about a page doesn’t grant permission to access it.”

Does the ChatGPT Search Cloudflare pilot help my pages get cited?

Not directly: the pilot can get pages into OpenAI’s index, but OpenAI evaluates and ranks those pages on its own, so discovery is no promise of a citation. Still, the direction is notable. For years, pages without backlinks were the hardest to get found. If infrastructure-level signals like freshness and real change detection become a discovery route for AI search, keeping practical, frequently updated pages accurate starts to matter more than building links to them.

Two caveats. Neither announcement says which sites participated or whether the new pages sit on those sites. And the companies acknowledge that finding new pages can increase crawling at first, so the promised load reduction may not show up immediately. Cloudflare has said it intends to open the capability more broadly later in 2026, but no date has been given.

Measurement is another weak spot. Search Engine Journal also reported that some ChatGPT links recently lacked both a UTM tag and a referrer, with a fix said to be rolling out, so attributing ChatGPT visits remains imperfect.

How to control OpenAI’s crawlers on your site

You control OpenAI’s access through two separate robots.txt user agents: OAI-SearchBot for ChatGPT search and GPTBot for content that may be used in training.

• Decide your split: allowing OAI-SearchBot while blocking GPTBot keeps you eligible for ChatGPT search results while signaling that crawled content shouldn’t be used for training. OpenAI uses this setup as an example on its crawler page.
• Know the trade-off: blocking OAI-SearchBot keeps you out of ChatGPT search results, though your site may still appear as a navigational link. Allowing both bots lets OpenAI use one crawl for both purposes.
• Use AI Crawl Control if you’re on Cloudflare: it is available on every plan and shows which AI services hit your site, with allow or block per crawler.
• Watch your logs: track OpenAI crawler hits for crawl spikes or wasted requests on redirects and missing files.
• Keep long-tail pages fresh: manuals, pricing, hours and requirements pages are exactly the type cited as valuable.

Sources

• Search Engine Journal: ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot