Cloudflare AI Bot Blocking: Complete Setup Guide (Free & Pro)

Updated: February 2025 | Works with Cloudflare Free, Pro, and Enterprise plans

Since July 2025, Cloudflare blocks AI bots by default for new sites. But if you set up your site earlier, you need to enable this protection manually. This guide shows you exactly how to use Cloudflare's AI bot blocking features, from the simple one-click option to advanced WAF rules.

---

Why Use Cloudflare for AI Bot Blocking?

Cloudflare offers the most effective AI bot protection because:

Advantage	Benefit
Edge blocking	Bots are stopped before reaching your server
Zero server load	No CPU/memory impact on your hosting
Always updated	Cloudflare maintains the bot list
Works on aggressive bots	Blocks bots that ignore robots.txt
Free tier available	Basic protection costs nothing

Cloudflare vs robots.txt

Method	Respects robots.txt	Ignores robots.txt
robots.txt	Blocked	Still gets through
Cloudflare	Blocked	Blocked

Cloudflare blocks ALL bots at the network level, regardless of whether they respect robots.txt.

---

Method 1: One-Click AI Bot Blocker (Easiest)

Cloudflare's built-in AI Scrapers and Crawlers blocker is the simplest solution.

Step-by-Step Setup

1. Log into Cloudflare Dashboard

   • Go to dash.cloudflare.com
   • Select your website

2. Navigate to Bot Settings

   • Click Security in the left sidebar
   • Click Bots

3. Find AI Scrapers Section

   • Scroll to "AI Scrapers and Crawlers"
   • You'll see a toggle switch

4. Enable Blocking

   • Toggle the switch to Block
   • Changes take effect immediately

What This Blocks

Cloudflare's AI bot blocker covers:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• Google-Extended (Google)
• Bytespider (ByteDance)
• CCBot (Common Crawl)
• PerplexityBot
• And 30+ other known AI crawlers

Verification

After enabling, check if it's working:

# This should return a Cloudflare block page
curl -A "GPTBot/1.0" https://yoursite.com/

---

Method 2: Custom WAF Rules (More Control)

For granular control over which bots to block, use Custom Rules.

When to Use Custom Rules

• Block specific bots while allowing others
• Add new bots not yet in Cloudflare's list
• Create exceptions for certain paths
• Log instead of block for monitoring

Creating a Custom Rule

1. Go to Security → WAF
2. Click Custom Rules
3. Click Create Rule

Rule: Block All AI Training Bots

Rule Name: Block AI Training Crawlers

Expression (Builder):

• Field: User Agent
• Operator: contains
• Value: GPTBot
• Add OR conditions for other bots

Expression (Edit expression):

(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "anthropic-ai") or
(http.user_agent contains "Google-Extended") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "Meta-ExternalAgent") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "ChatGLM") or
(http.user_agent contains "Sogou") or
(http.user_agent contains "360Spider") or
(http.user_agent contains "DeepSeekBot") or
(http.user_agent contains "xAI-Grok")

Action: Block

Deploy the rule.

Rule: Block AI Bots But Allow Search

Allow AI search engines but block training bots:

Expression:

(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "anthropic-ai") or
(http.user_agent contains "Google-Extended") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "Meta-ExternalAgent") or
(http.user_agent contains "Bytespider")

This blocks training bots but allows:

• PerplexityBot (AI search)
• ChatGPT-User (assistant)
• YouBot (AI search)

---

Method 3: Bot Fight Mode (Free)

Even without the AI-specific blocker, Cloudflare's Bot Fight Mode helps.

Enable Bot Fight Mode

1. Go to Security → Bots
2. Find Bot Fight Mode
3. Toggle to On

What Bot Fight Mode Does

• Challenges suspicious automated traffic
• Uses JavaScript challenges to verify browsers
• May catch some AI bots that spoof user agents

Note: This is less targeted than the AI bot blocker but adds an extra layer of protection.

---

Method 4: Super Bot Fight Mode (Pro Plan)

Cloudflare Pro ($20/month) unlocks additional options.

Pro Features

1. Definitely Automated - Block requests from known bot signatures
2. Likely Automated - Challenge suspicious requests
3. Verified Bots - Allow/block known good bots separately

Recommended Pro Settings

Category	Setting	Reason
Definitely Automated	Block	Stop obvious bots
Likely Automated	Managed Challenge	Filter suspicious traffic
Verified Bots	Allow	Keep search engines working
AI Scrapers	Block	Stop AI training bots

---

Method 5: Firewall Rules (Legacy)

If you have existing Firewall Rules, you can add AI bot blocking there.

Creating a Firewall Rule

1. Go to Security → WAF → Firewall Rules (if available)
2. Click Create a Firewall Rule

Rule:

• Name: Block AI Crawlers
• Expression: (cf.client.bot) and (http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot")
• Action: Block

Note: Firewall Rules are being migrated to Custom Rules. Use Custom Rules for new setups.

---

Advanced: Combining Multiple Methods

For maximum protection, layer your defenses:

Recommended Stack

Layer 1: Cloudflare AI Scrapers Blocker (catches known bots)
         ↓
Layer 2: Custom WAF Rule (catches new/custom bots)
         ↓
Layer 3: Bot Fight Mode (catches spoofed bots)
         ↓
Layer 4: Rate Limiting (prevents aggressive crawling)

Setting Up Rate Limiting

Block IPs that make too many requests:

1. Go to Security → WAF → Rate Limiting Rules
2. Create new rule:

Expression:

(http.request.uri.path contains "/")

Characteristics:

• Requests: 100
• Period: 1 minute
• Action: Block for 1 hour

This stops aggressive crawlers even if they change user agents.

---

Monitoring & Analytics

Check Your Blocking Stats

1. Go to Security → Events
2. Filter by:
   • Action: Block
   • Service: WAF / Bot Fight Mode

What to Look For

• User Agent - Confirm AI bots are being blocked
• IP Address - Identify repeat offenders
• Country - Many aggressive bots come from specific regions
• Request Count - Measure bandwidth saved

Export Reports

1. Click Export on the Security Events page
2. Choose CSV or JSON format
3. Analyze in your preferred tool

---

Troubleshooting

"AI bots are still getting through"

1. Check rule order - Rules process top to bottom
2. Verify expression syntax - Use the expression tester
3. Check for user agent variations - Some bots use different strings
4. Enable logging first - Switch action to "Log" to debug

"Legitimate traffic is being blocked"

1. Check Security Events for false positives
2. Add exceptions for:
   • Known good IPs
   • Specific paths (like /api/)
   • Verified services

Exception rule example:

(http.user_agent contains "GPTBot") and
not (ip.src eq 192.168.1.1)

"I don't see the AI Scrapers option"

• Feature may be rolling out gradually
• Try clearing browser cache
• Contact Cloudflare support if missing

"Changes aren't taking effect"

• Cloudflare propagation takes 1-2 minutes
• Clear your browser cache
• Try testing from a different network

---

Cloudflare Plans Comparison

Feature	Free	Pro ($20/mo)	Business ($200/mo)
AI Scrapers Blocker	Yes	Yes	Yes
Custom WAF Rules	5 rules	20 rules	100 rules
Bot Fight Mode	Basic	Super	Super
Rate Limiting	Basic	Advanced	Advanced
Bot Analytics	Limited	Detailed	Full

For most websites: Free plan with AI Scrapers Blocker is sufficient.

For high-traffic sites: Pro plan adds better analytics and more rule capacity.

---

Best Practices

Do's

• Enable AI Scrapers Blocker immediately
• Monitor Security Events weekly
• Keep custom rules updated with new bots
• Test changes with "Log" action first
• Allow legitimate AI assistants if desired

Don'ts

• Don't block all bots (you'll break search engines)
• Don't skip the built-in AI blocker for custom rules alone
• Don't forget to verify changes are working
• Don't ignore the analytics data

---

FAQ

Does this affect my SEO?

No. Cloudflare's AI bot blocking specifically targets AI training and scraping bots. Traditional search engine crawlers (Googlebot, Bingbot) are not affected.

Will this block ChatGPT from answering questions about my site?

It depends. The AI Scrapers Blocker may block ChatGPT-User (the assistant bot). If you want ChatGPT to cite your content:

1. Use custom rules instead of the one-click blocker
2. Only block training bots (GPTBot), not assistant bots (ChatGPT-User)

Can bots bypass Cloudflare?

Some sophisticated bots might try to:

• Spoof user agents
• Use residential proxies
• Bypass Cloudflare entirely

Mitigation:

• Enable Bot Fight Mode for additional protection
• Use rate limiting to catch aggressive behavior
• Monitor for unusual traffic patterns

How much bandwidth will I save?

Typical savings after enabling AI bot blocking:

• Small sites: 20-40%
• Medium sites: 30-50%
• Large content sites: 50-70%

Check your Cloudflare Analytics before and after to measure the impact.

---

Conclusion

Cloudflare provides the most effective AI bot protection available, and the basic features are free. Whether you use the one-click AI Scrapers Blocker or custom WAF rules, you can stop AI crawlers before they reach your server.

Quick Start:

1. Log into Cloudflare
2. Go to Security → Bots
3. Enable "AI Scrapers and Crawlers" blocking
4. Done!

For custom control, add WAF rules targeting specific bots you want to block or allow.

---

Want to check if your site is properly protected? Use our free AI crawler checker to verify which bots can still access your content.