Blog & Practical Guides

Practical articles and technical notes on how AI crawlers interact with WordPress sites, how to interpret AI referral traffic, and how to configure robots.txt, llms.txt, and structured data without guesswork.

Latest Articles

Training Crawlers vs. AI Search Bots in WordPress: A robots.txt Guide

Many WordPress site owners want to opt out of foundation model training while keeping their pages visible in ChatGPT Search, Perplexity, and Google AI Overviews. Learn the difference between model training crawlers (GPTBot, ClaudeBot, Google-Extended), AI search indexing bots (OAI-SearchBot, PerplexityBot), and user-triggered fetchers (ChatGPT-User), with copy-ready robots.txt templates and four common WordPress pitfalls to avoid.

• Read the full guide →

---

Tools & Documentation

• AI Search Readiness Checker: Run a live check on your domain’s robots.txt, Schema.org structured data, and llms.txt.
• Getting Started with OwlyCrackGeo: Install the free, self-hosted WordPress plugin to log 60+ AI crawlers and compare bot hits with real human referrals.
• Understanding Your Data: Learn what crawler logs, referral clicks, readiness scores, and IndexNow notifications can—and cannot—tell you.