Pay Per Crawl

July 1, 2026

Making AI search smarter

Search is how we find nearly everything on the web creators, merchants, answers. AI is rewriting the rules, leaving creators caught between staying discoverable in an agentic era and getting paid for their work. Today we're launching two initiatives to help.

September 26, 2025

An AI Index for all our customers

Cloudflare will soon automatically create an AI-optimized search index for your domain, and expose a set of ready-to-use standard APIs and tools including an MCP server, LLMs.txt, and a search API.

August 28, 2025

The next step for content creators in working with AI bots: Introducing AI Crawl Control

Cloudflare launches AI Crawl Control (formerly AI Audit) and introduces easily customizable 402 HTTP responses.

July 1, 2025

Content Independence Day: no AI crawl without compensation!

Its Content Independence Day: Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for content.

July 1, 2025

Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

July 1, 2025

Message Signatures are now part of our Verified Bots Program, simplifying bot authentication

Bots can start authenticating to Cloudflare using public key cryptography, preventing them from being spoofed and allowing origins to have confidence in their identity.

July 1, 2025

Control content use for AI training with Cloudflares managed robots.txt and blocking for monetized content

Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf.

July 1, 2025

The crawl before the fall of referrals: understanding AIs impact on content providers

Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This helps site owners make decisions about which AI bots to allow or block.

July 1, 2025

From Googlebot to GPTBot: whos crawling your site in 2025

From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%.