Somebody pasted a “block AI” robots.txt snippet into your WordPress site last week. It felt responsible. GPT in the user-agent name? Disallow. Claude? Disallow. Anything that smells like a model? Out.
Then a prospect asked ChatGPT which backup plugin to trust, or Perplexity which SEO stack survives AI Overviews, and your competitor got named. You got silence. Search Console still looked fine. Your titles were still intact. You just opted out of the citation path without noticing.
That is the trap. Training crawlers and citation crawlers share a family resemblance and do different jobs. Rankify is built around that split: sixteen crawlers across ten vendors, each labelled before you choose, robots.txt you can actually reason about, and an llms.txt map that costs nothing to serve. This post is about keeping the trainers you refuse from taking the citers with them.
Training vs citation is a real policy line
A training crawler gathers text to improve a model. You get no byline and no click. Blocking it is a rights and preference decision. On Rankify’s product labelling, GPTBot and CCBot sit in that bucket: they train on content rather than sending a reader back to you.
A citation crawler fetches pages so an assistant can answer a live question and name sources. Block those and you disappear from the answers, and from the traffic they send. Rankify labels OAI-SearchBot as the ChatGPT citation path and PerplexityBot as the Perplexity citation path. Same product family as the trainers. Opposite commercial outcome for your site.
Most generic “block AI” checklists flatten that into one Deny-All. Rankify refuses that flattening on purpose. Training vs citation is separated and labelled in the crawler list so you can allow assistants that cite while blocking ones that only train.
If you are unsure, allow everything. The traffic cost of being crawled is trivial. Being absent from AI answers is a slow, invisible loss. You notice it when someone else is the named source.
What the Allow AI Crawlers toggle actually does
On Rankify → Settings → AI & GEO, Allow AI Crawlers writes GPTBot, ClaudeBot, PerplexityBot, and Google-Extended into your robots.txt in one move.
Turning that toggle off is not a “pause training only” switch. Docs are blunt: it blocks AI training and retrieval crawlers, and also removes you from AI answers that cite sources. If your goal is to be quoted, blocking crawlers is the one setting guaranteed to prevent it. Blocking is a reasonable choice when your objection is training, but citation goes with the blanket.
Use the toggle when you truly want a hard no to the whole AI fetch layer. Use the per-crawler list when the blanket is too blunt.
Per-crawler control when one switch is too crude
The per-crawler list is the fine dial. Sixteen crawlers, ten vendors, each one explained in the UI before you flip Allowed or Blocked. That is how you keep OAI-SearchBot and PerplexityBot reading while GPTBot and CCBot stay out, if that is your policy.
You do not need to memorise vendor IP charts to start. You need to stop treating every string with “Bot” in it as the same threat. Write the policy you mean:
- Want maximum chance of being quoted? Allow citation crawlers. Leave trainers allowed unless you have a clear objection.
- Want to refuse training datasets but stay citeable? Block the labelled trainers. Keep the labelled citers Allowed.
- Want off the AI web entirely? Turn Allow AI Crawlers off and accept the citation cost.
Rankify writes those choices into robots.txt for you. No hand-edited mystery file that drifts after the next plugin update.
Publish llms.txt even when you block some trainers
Access is only half of reachability. Models increasingly look for /llms.txt: a plain-text map of important pages with short summaries, written for language models rather than browsers.
Rankify publishes it for free. It is on by default. Cost to serve is near zero. It is not a substitute for allowing the citation crawlers, and it is not a magic rank boost. It is a readable index so a model that is allowed to fetch you can find the pages you actually care about.
If you already blocked every AI agent in robots.txt, llms.txt will not save you. Fix access first, then keep the map on.
Measure the funnel: fetch is not citation
We already wrote about how to tell if AI answer engines are using your WordPress pages. The short version for this policy post:
- Access. Are citation crawlers Allowed?
- Reads. Which assistants fetch which URLs?
- Referrals. Did a person arrive from an answer?
- Structure. Are passages quotable once someone reads them?
The AI crawler log covers the middle of that funnel. Free keeps seven days. Pro extends to ninety and verifies each crawler’s identity by reverse DNS or the vendor’s published IP ranges. A user-agent string is trivial to spoof. Without verification, anything can inflate your GPTBot line. Verified identity is the difference between a number you can trust and a vanity counter.
Pro AI referral tracking counts visits from ChatGPT, Perplexity, Claude, Gemini, and Copilot per page. A crawler read proves reachability. A person arriving from an answer proves a citation (or at least a linked mention) that sent a human. Most analytics bury that under generic referral traffic or lose it when the referrer is stripped.
Read it as a funnel. Crawlers reading but no referrals means you are reachable but not quotable. No crawlers at all means an access problem. Check robots.txt before you rewrite the entire blog.
Make the pages worth quoting after you allow the bots
Allowing OAI-SearchBot does not force ChatGPT to name you. Structure still matters.
On Pro, Rankify’s answer blocks find question-shaped headings and emit FAQPage markup plus a concise answer summary, which is the format answer engines quote. That works with headings you already wrote. You do not need a separate FAQ plugin just to mark the Q&A shape.
Pro IndexNow tells Bing, Yandex, and Seznam the moment a page is published or changed. Google does not take part in IndexNow. Its Indexing API only accepts job postings and live streams, so Rankify does not pretend otherwise. Bing still matters here beyond classic search share, because its index feeds several AI assistants.
Rankify does not scrape Google rankings. No honest WordPress plugin can know position without an outside rank API or a scraper that will get you blocked. It reports what it can observe from your own site: coverage, broken URLs, which AI assistants read you, and which ones sent someone back.
If you already run Yoast, Rank Math, All in One SEO, or SEOPress, Rankify reads those titles, descriptions, canonicals, focus keywords, and robots settings in place. Nothing is copied, moved, or deleted. That is the same migration story we covered when we explained how Rankify reads Yoast, then shows whether ChatGPT sent traffic. You can fix crawler policy without ripping titles out first.
A practical robots policy for WordPress owners
Start with intent, not fear.
Step 1. Open Rankify → AI & GEO. Confirm Allow AI Crawlers is on unless you truly want off the citation web.
Step 2. Open the per-crawler list. Allow every crawler labelled for citation if you want to be quoted. Block labelled trainers only if that is a deliberate company policy, not a Twitter panic.
Step 3. Leave llms.txt on. Pick the post types that belong in the map. Exclude noindex the same way you would from a sitemap.
Step 4. Watch the crawler log for a week. If citation crawlers never appear after an allow, check host-level firewall rules, CDN bot fights, and competing robots plugins before you blame the content.
Step 5. On Pro, watch AI referrals per page. Optimize the URLs that get reads but no arrivals: shorter sections, question headings, answer in the first paragraph. That is writing work. The plugin cannot invent substance.
Do not paste a random “block all GPT*” gist from a forum into robots.txt and walk away. That gist almost certainly collapses training and citation into one Disallow, which is exactly how sites erase themselves from ChatGPT answers while congratulating themselves on “protecting content.”
What you are buying with Rankify (and the membership)
Rankify is the SEO plus answer-engine layer: titles, schema, sitemaps, redirects, robots.txt, llms.txt, labelled AI crawler control, and the crawler log on free. Pro adds verified crawler identity, ninety-day history, AI referral tracking, answer blocks, retrieval score, IndexNow, and the rest of the answer-engine measurement stack.
It is not a second SEO subscription bolted on the side. Pro is sold as one DaftPlug membership. One licence unlocks Rankify with Progressify, Lightify, Generatify, and Snapshotify. Plans differ by site count, not by feature list. See pricing for current tiers, Freemius checkout, and the 14-day money-back guarantee. New Pro plugins join existing memberships. You do not buy a fifth line item every time the suite grows.
Bottom line
Blocking AI is not a single checkbox. Training crawlers gather text for models. Citation crawlers are how ChatGPT, Claude, and Perplexity find pages to quote. Collapse those into one robots deny and you can refuse training while also deleting yourself from the answers that send people.
Use Rankify’s labelled list of sixteen crawlers across ten vendors. Keep the Allow AI Crawlers toggle on unless you mean a full opt-out. Publish llms.txt. Measure fetches, then referrals. Fix structure after access. That is how you block trainers on purpose without erasing yourself from ChatGPT answers by accident.