How to Block AI Crawlers on WordPress: 5 Easy Methods (2025)

Updated: February 2025 | Works with WordPress 6.x and all major themes

WordPress powers over 40% of the web, making it a prime target for AI crawlers like GPTBot, ClaudeBot, and Bytespider. If you're running a WordPress site and want to protect your content from being scraped for AI training, this guide covers 5 proven methods from beginner-friendly plugins to advanced configurations.

---

Why WordPress Sites Need AI Crawler Protection

AI companies are aggressively crawling WordPress sites because:

• Easy to identify: WordPress has predictable URL structures
• Content-rich: Blogs, news sites, and e-commerce stores have valuable data
• Often unprotected: Many site owners don't know about AI crawlers

The Impact on Your WordPress Site

Problem	Impact
Bandwidth costs	AI bots can consume 50-70% of your bandwidth
Server load	Shared hosting may throttle your site
Content theft	Your articles train AI models without permission
SEO competition	AI-generated content competes with your originals

---

Method 1: WordPress Plugins (Easiest)

Option A: Block AI Crawlers Plugin (Free)

The simplest solution for non-technical users.

Installation:

1. Go to Plugins → Add New
2. Search for "Block AI Crawlers"
3. Click Install Now → Activate

What it does:

• Automatically updates your robots.txt
• Adds noai, noimageai meta tags
• Blocks 20+ known AI crawlers

Pros: One-click setup, no configuration needed
Cons: Only uses robots.txt (aggressive bots may ignore)

Option B: Bot Traffic Shield (Recommended)

More comprehensive protection with analytics.

Features:

• Pre-configured list of 40+ AI crawlers
• Real-time blocking (not just robots.txt)
• Detailed logs of blocked attempts
• IP address and user agent tracking

Installation:

1. Go to Plugins → Add New
2. Search for "Bot Traffic Shield"
3. Install and activate
4. Go to Settings → Bot Traffic Shield
5. Enable "Block AI Crawlers" option

Option C: LLM Bot Tracker (Analytics Focus)

Best for monitoring before blocking.

Use this if you want to:

• See which AI bots are crawling your site
• Measure bandwidth impact
• Make data-driven blocking decisions

---

Method 2: Manual robots.txt (Intermediate)

Edit your robots.txt file directly for precise control.

Access Your robots.txt

Option 1: Yoast SEO Plugin

1. Go to SEO → Tools → File Editor
2. Edit the robots.txt section

Option 2: Rank Math Plugin

1. Go to Rank Math → General Settings → Edit robots.txt

Option 3: Direct File Access

1. Connect via FTP/SFTP
2. Navigate to your WordPress root directory
3. Create or edit robots.txt

Recommended robots.txt Rules

# Block AI Training Crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Disallow: /

# Allow regular search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Block ALL 42 AI Crawlers

For complete protection, add rules for all known AI crawlers:

# === AI Training Bots ===
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Bytespider
User-agent: CCBot
User-agent: Baiduspider
User-agent: Sogou
User-agent: 360Spider
User-agent: ChatGLM-Spider
User-agent: DeepSeekBot
User-agent: cohere-ai
User-agent: PanguBot
User-agent: xAI-Grok
Disallow: /

# === AI Search Bots ===
User-agent: PerplexityBot
User-agent: YouBot
User-agent: OAI-SearchBot
User-agent: Kangaroo Bot
User-agent: iaskspider
Disallow: /

# === AI Assistants ===
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Perplexity-User
Disallow: /

---

Method 3: Security Plugins (Advanced)

Use your existing security plugin for AI bot blocking.

Wordfence (Most Popular)

1. Go to Wordfence → Firewall → Blocking
2. Click Create Blocking Rule
3. Set Block Type to "Custom Pattern"
4. Enter user agent patterns:

GPTBot
ClaudeBot
Bytespider
CCBot

1. Set action to "Block"
2. Save the rule

Sucuri Security

1. Go to Sucuri Security → Firewall (WAF)
2. Navigate to Access Control
3. Add user agent blocks for AI crawlers

All In One WP Security

1. Go to WP Security → Firewall
2. Enable "Block Bad Bots"
3. Add custom user agent rules

---

Method 4: .htaccess Rules (Server-Level)

For aggressive bots that ignore robots.txt, use .htaccess blocking.

Access .htaccess

Method A: File Manager (cPanel)

1. Log into cPanel
2. Open File Manager
3. Navigate to your WordPress root
4. Show hidden files (Settings)
5. Edit .htaccess

Method B: FTP/SFTP

1. Connect to your server
2. Download .htaccess
3. Edit locally
4. Upload back

Add These Rules

Add this code before the # BEGIN WordPress line:

# Block AI Crawlers
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} anthropic [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Google-Extended [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalAgent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ChatGLM [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sogou [NC,OR]
RewriteCond %{HTTP_USER_AGENT} 360Spider [NC]
RewriteRule .* - [F,L]
</IfModule>
# End Block AI Crawlers

Verify It Works

After saving, test with:

curl -A "GPTBot" https://yoursite.com/

You should see a 403 Forbidden response.

---

Method 5: Cloudflare Integration (Best Protection)

If you use Cloudflare (even the free plan), you can block AI crawlers at the edge.

Enable Built-in AI Bot Blocker

1. Log into Cloudflare Dashboard
2. Select your WordPress site
3. Go to Security → Bots
4. Find AI Scrapers and Crawlers
5. Toggle to Block

This single setting blocks all known AI crawlers before they reach your server!

Custom WAF Rules (Pro Feature)

For more control, create a custom rule:

1. Go to Security → WAF → Custom Rules
2. Create new rule:

Expression: (http.user_agent contains "GPTBot") or
            (http.user_agent contains "ClaudeBot") or
            (http.user_agent contains "Bytespider")
Action: Block

---

Comparison: Which Method Should You Use?

Method	Difficulty	Effectiveness	Best For
Plugins	Easy	Medium	Beginners
robots.txt	Easy	Medium	Basic protection
Security plugins	Medium	High	Existing users
.htaccess	Hard	High	Tech-savvy users
Cloudflare	Easy	Highest	Everyone with Cloudflare

Our Recommendation

For most WordPress users:

1. Install "Bot Traffic Shield" plugin
2. Add robots.txt rules via Yoast/Rank Math
3. If using Cloudflare, enable AI bot blocking

For maximum protection:

1. Use Cloudflare's AI bot blocker
2. Add .htaccess rules as backup
3. Monitor with LLM Bot Tracker

---

Common Issues & Troubleshooting

"My robots.txt changes aren't working"

• Clear any caching plugins (WP Super Cache, W3 Total Cache)
• Wait 24-48 hours for crawlers to re-check
• Verify at yoursite.com/robots.txt

"Bots are still getting through"

Some bots (Bytespider, Sogou) ignore robots.txt. Use:

• .htaccess blocking
• Cloudflare WAF
• Security plugin firewall

"My site is slower after adding rules"

• .htaccess rules process on every request
• Consider Cloudflare for edge-level blocking
• Use caching plugins to offset the load

"I blocked too much and broke something"

If search engines can't crawl your site:

1. Check robots.txt for User-agent: * rules
2. Make sure Googlebot and Bingbot are allowed
3. Test with Google Search Console

---

Monitoring Your Protection

After implementing blocking:

1. Check server logs for AI bot requests
2. Use CheckAIBots.com to verify your robots.txt
3. Monitor bandwidth for reduction (expect 30-70% drop)
4. Review analytics for traffic changes

---

FAQ

Will blocking AI crawlers hurt my SEO?

No. AI crawlers (GPTBot, ClaudeBot) are separate from search engine crawlers (Googlebot, Bingbot). Blocking AI bots has zero impact on your search rankings.

Should I block ChatGPT-User and Claude-Web?

These are "assistant" bots that fetch pages when users ask questions. Blocking them:

• Pro: Prevents any AI from accessing your content
• Con: Users can't get answers that cite your site

Most sites allow assistant bots but block training bots.

Do I need all 5 methods?

No! Choose based on your situation:

• Basic protection: Plugin + robots.txt
• Maximum protection: Cloudflare + .htaccess + plugin

Which AI bots ignore robots.txt?

Known offenders:

• Bytespider (ByteDance)
• 360Spider
• Sogou
• ChatGLM-Spider
• xAI-Grok (sometimes disguises user agent)

For these, you MUST use server-level blocking.

---

Conclusion

Protecting your WordPress site from AI crawlers is essential in 2025. Whether you choose a simple plugin or implement multiple layers of protection, the key is to act now before more of your content ends up in AI training datasets.

Next steps:

1. Check your site's current AI bot exposure
2. Implement at least one blocking method today
3. Monitor and adjust as new bots emerge

---

Need to check if your WordPress site is already being crawled? Use our free AI crawler checker to see which bots can access your content.