Training Crawlers vs. AI Search Bots in WordPress

Many WordPress site owners want to prevent their articles from being scraped for foundation model training, yet still want their pages cited—and clicked—in ChatGPT Search, Perplexity, and Google AI Overviews.

Blocking every AI user agent with a single blanket rule often cuts off search and citation bots along with training crawlers. Understanding the three categories of AI bots helps you set robots.txt rules that match your publishing goals.

Three roles of AI user agents

AI providers increasingly separate their user agents by purpose:

Category	Representative User Agents	Primary Purpose	Impact of Blocking in robots.txt
1. Model training crawlers	GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider	Collect public web text to train or fine-tune generative models	Signals that your content should not be used for future model training
2. AI search & indexing bots	OAI-SearchBot, PerplexityBot, Applebot	Discover and index pages to surface links and citations in AI search answers	Reduces or prevents your pages from appearing as cited sources in search results
3. User-triggered fetchers	ChatGPT-User, Claude-Web, Perplexity-User	Fetch a specific URL when a human user asks an AI assistant to read or summarize a link	Prevents the assistant from reading the page during a live user session

Important distinction: robots.txt is a voluntary protocol. Reputable providers document which user agents honor Disallow directives, but uncooperative scrapers can ignore robots.txt or spoof User-Agent headers.

Common WordPress robots.txt strategies

Strategy A: Allow AI search citations, opt out of model training

If you want your WordPress site to appear in AI search results and receive referral clicks, while signaling an opt-out from foundation model training:

# Allow OpenAI search indexing, opt out of model training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

# Allow Perplexity search indexing
User-agent: PerplexityBot
Allow: /

# Opt out of Anthropic & Google model training
User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Keep standard WordPress admin paths closed
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Strategy B: Maximize AI discovery and visibility

For documentation sites, open-source projects, and marketing sites where broad visibility across both AI search and model knowledge is desirable:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Four WordPress pitfalls to check

1. Virtual vs. physical robots.txt WordPress dynamically generates /robots.txt unless a physical robots.txt file exists in your web root. If both an SEO plugin and a physical file exist, the web server serves the physical file and ignores plugin changes.
2. Overlapping SEO plugin rules Plugins such as Yoast SEO, Rank Math, or All in One SEO may inject their own User-agent blocks. Always inspect the final rendered output at https://your-domain.com/robots.txt.
3. CDN and WAF bot challenges Even if your robots.txt allows OAI-SearchBot or PerplexityBot, a CDN firewall or “Bot Fight Mode” can return 403 Forbidden or a JavaScript challenge before the request ever reaches WordPress.
4. Crawler visits do not equal reader referrals Seeing thousands of GPTBot or ClaudeBot hits in your server logs only means pages were fetched. Compare crawler requests against actual human referral clicks from ChatGPT, Perplexity, and other AI tools to see which pages bring readers.

Verify your setup

1. Test your live robots.txt and llms.txt externally: Run a free scan with the AI Search Readiness Checker to see how major AI user agents evaluate your public headers, robots.txt, and Schema.org markup.
2. Observe real bot traffic inside WordPress: Install the free OwlyCrackGeo plugin on WordPress.org to log visits across 60+ AI crawlers, verify supported bots via forward-confirmed reverse DNS, and compare crawler hits with real AI referral clicks.
3. Understand what each signal means: Read Understanding your data to see how crawler activity, referral attribution, content readiness scores, and IndexNow notifications fit together.