[ Web Proxy ]

Viewing: https://developers.cloudflare.com/ai-crawl-control/get-started/

[Back] [Original]

Get started with Cloudflare AI Crawl Control Cloudflare AI Crawl Control docsSkip to content

[]DocsDirectory API SDKs Changelog

SearchCtrlK

AI Crawl Control

Overview
Get started
Features
- Analyze AI traffic
- Manage AI crawlers
- Directives
- Pay Per CrawlBeta
  What is Pay Per Crawl?
  Use pay per crawl as a site owner
  Enable in account settings
  Set a pay per crawl price
  Select crawlers to charge
  Monitor activity
  Manage payouts
  Advanced configuration
  Use pay per crawl as an AI owner
  Set up your account
  Verify your AI crawler
  Connect Stripe
  Discover payable content
  Crawl pages
  Error codes
  Pay Per Crawl FAQ
Configuration
Reference
Changelog
Agent resources

Home
/AI Crawl Control
/Get started

Get started

Last updated Apr 23, 2026Copy as MarkdownView as MarkdownAgent setup

OverviewPrerequisites1. Monitor AI crawler activity at a glance2. Block specific AI crawlers3. Explore detailed metricsPlan comparisonNext stepsRelated resources

This guide instructs you on how to:

View AI crawlers that are interacting with pages in your domain (a Cloudflare zone).
Use AI Crawl Control to block individual crawlers from accessing your content.

Prerequisites

Sign up for a Cloudflare account .
Connect your domain to Cloudflare.
Make sure your domain is proxying traffic through Cloudflare.

1. Monitor AI crawler activity at a glance

Go to AI Crawl Control.
Go to AI Crawl Control ↗
Review the snapshot of your AI crawler activity in the Overview tab.
Use the filters to view activity by specific date ranges, crawlers, operators, hostnames, or paths.

2. Block specific AI crawlers

To block specific AI crawlers:

Go to AI Crawl Control.
Go to AI Crawl Control ↗
Review which AI crawlers are accessing your domain in the Crawlers tab.
In the Action column, select Block.

Quality of AI crawler detection

On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings . This enables AI Crawl Control to detect well-known, self-identifying AI crawlers.

Upgrade your plan to enable a more thorough detection using Cloudflare's Bot Management detection ID field.

To block specific AI crawlers:

Go to AI Crawl Control.
Go to AI Crawl Control ↗
Review which AI crawlers are accessing your domain in the Crawlers tab.
In the Action column, select Block.

For more information, refer to Manage AI crawlers.

You can also create more complex rules when taking action on AI crawlers, using Cloudflare WAF. For more information on creating more specific rules, refer to Create a custom rule in the dashboard.

3. Explore detailed metrics

For more detailed analytics, use the Metrics tab.

Go to AI Crawl Control.
Go to AI Crawl Control ↗
Review detailed breakdowns by date range, crawler, operator, status code, hostname, or path in the Metrics tab.

Note that on free plans, the Metrics tab only displays metrics for the past 24 hours.

Plan comparison

All plans	Enterprise plans with Bot Management
AI crawler detection via user agent strings	Advanced AI crawler detection via Bot Management detection ID
Maximum 24-hour analytics window	Configurable analytics timeframes
Allow/block controls	Allow/block controls, and the ability to charge AI crawlers using pay per crawl