Choose Which AI Crawlers May Read Your WordPress Site, One Vendor at a Time

Somebody pasted a “block all AI” robots.txt snippet into WordPress last month. It felt responsible. Anything with Bot in the name got Disallow.

Then a prospect asked ChatGPT which plugin to trust, or Perplexity which host to pick, and a competitor got named. You got silence. Search Console still looked fine. You had opted out of citation without noticing.

That is the trap. Training crawlers and citation crawlers look related and do different jobs. Rankify is built around that split: sixteen crawlers across ten vendors, each explained in the UI, with training vs citation labelled separately. Blocking citation crawlers removes you from AI answers. This post is the per-vendor control panel, not a second deep dive on llms.txt (we already covered publishing that map).

If you want the companion map for models that are allowed to fetch you, keep Publish an llms.txt map so AI models can find what to cite on WordPress nearby. Access first. Map second.

What Rankify actually ships for AI crawler control

Feature list for this article:

• AI crawler control for sixteen crawlers across ten vendors
• Each crawler explained in the UI before you choose
• Training vs citation labelled separately
• Choices written into robots.txt for you
• Related Rankify pieces: llms.txt (mention only here), AI crawler log, AI referral tracking (ChatGPT / Perplexity / Claude / Gemini / Copilot), plus the broader SEO toolkit (title templates, schema @graph, redirects/404, content analysis, answer blocks, Product schema, IndexNow optional, Yoast/Rank Math data read in place)

No invented rank tracker. No claim that blocking trainers automatically boosts Google rankings. This is access policy.

Why one “block AI” switch is too blunt

A training crawler gathers text to improve a model. You get no byline and no click. Blocking it is a rights and preference decision.

A citation crawler fetches pages so an assistant can answer a live question and name sources. Block those and you disappear from the answers, and from the traffic they send.

Most generic checklists flatten that into Deny-All. Rankify refuses that flattening on purpose. Training vs citation stays labelled so you can allow assistants that cite while blocking ones that only train, if that is your policy.

Write the policy you mean:

• Want maximum chance of being quoted? Allow citation crawlers. Leave trainers allowed unless you have a clear objection.
• Want to refuse training datasets but stay citeable? Block the labelled trainers. Keep the labelled citers Allowed.
• Want off the AI web entirely? Block broadly and accept the citation cost.

Guessing from user-agent folklore is how good sites vanish from answers by accident.

How to set per-crawler access in wp-admin

1. Open Rankify’s AI crawler controls

In wp-admin, open Rankify and find the AI / GEO crawler access settings. You should see the list of crawlers with explanations and training vs citation labels, plus Allowed / Blocked controls per entry.

Skim the whole list once before flipping anything. Sixteen entries across ten vendors is manageable if you read the labels.

2. Decide your default posture

If you are unsure, allow everything first. The traffic cost of being crawled is usually trivial. Being absent from AI answers is a slow, invisible loss. You notice it when someone else is the named source.

If legal or brand policy forbids training use, start by blocking labelled trainers only. Leave citation crawlers Allowed unless policy says otherwise.

3. Set each vendor on purpose (not by vibe)

Work vendor by vendor. For each crawler:

1. Read the explanation Rankify shows.
2. Note whether it is labelled training or citation.
3. Set Allowed or Blocked to match your written policy.
4. Move to the next. Do not batch-flip without reading.

You do not need to memorise vendor IP charts to start. You need to stop treating every string with “Bot” in it as the same threat.

4. Confirm robots.txt reflects the choices

Rankify writes the decisions into robots.txt. Open your site’s robots.txt after saving and spot-check that the Disallow / Allow lines match what you set. No hand-edited mystery file that drifts after the next plugin update.

If another SEO plugin also rewrites robots.txt, resolve ownership. Two writers on one file is how policies silently reverse.

5. Keep llms.txt in mind (briefly)

llms.txt is a plain-text map of important pages for language models. Rankify can publish it. It is not a substitute for allowing citation crawlers. If you blocked every AI agent in robots.txt, the map will not save you. Fix access first, then keep the map on for models that are allowed to fetch.

6. Verify with crawler log and referral tracking

After policy settles:

• Use the AI crawler log to see which assistants fetch which URLs.
• Use AI referral tracking for visits from ChatGPT, Perplexity, Claude, Gemini, and Copilot per page.

Read it as a funnel. Crawlers reading but no referrals means you are reachable but not quotable yet. No crawlers at all means an access problem. Check robots.txt before you rewrite the entire blog.

What good looks like after a week

A healthy setup feels boring:

• Citation crawlers you care about show as Allowed, with matching robots.txt lines.
• Labelled trainers you refuse stay Blocked without collateral damage to citers.
• The AI crawler log shows verified fetches on the pages you actually want quoted (money pages, docs, comparisons), not only tag archives.
• AI referral tracking starts to show occasional human arrivals from ChatGPT, Perplexity, Claude, Gemini, or Copilot.
• llms.txt is live for allowed fetchers, but you did not treat it as a substitute for access.
• No second plugin silently rewrote robots.txt overnight.

If the log is empty on citation bots you Allowed, check CDN WAF rules and host bot blockers next. Rankify can open the door in robots.txt. A firewall can still slam it.

If referrals stay flat while crawlers read, the access problem is solved and the quotability problem remains. Then lean on Rankify’s content analysis, FAQ/QAPage answer blocks, Product schema, and clearer titles, not another Disallow.

Pitfalls (the ones that bite)

Pasting a viral “block all AI” snippet. It often kills citation paths you still wanted.

Blocking citation crawlers while complaining about “no AI traffic.” You opted out.

Ignoring labels and sorting by vendor logo vibes. Training vs citation is the policy line.

Editing robots.txt by hand on top of Rankify. Drift and duplicate rules follow.

Publishing llms.txt while disallowing every fetcher. Map without access is decoration.

Never checking the crawler log. Spoofable user-agents exist; treat logs as a signal, and use Rankify’s verification where available on Pro rather than vanity counters alone.

Expecting crawler policy to replace on-page SEO. Titles, schema, FAQ answer blocks, and content quality still decide whether a fetched page is worth quoting.

Practical checklist before you call it done

1. Write a one-paragraph policy: trainers vs citers.
2. Open Rankify crawler controls and read all sixteen explanations.
3. Set Allowed / Blocked per crawler to match the policy.
4. Confirm robots.txt matches.
5. Resolve conflicts if another plugin also writes robots.txt.
6. Keep llms.txt available for allowed fetchers (do not treat it as the main fix).
7. Watch AI crawler log for a week.
8. Watch AI referral tracking for human arrivals from assistants.
9. Revisit after major vendor changes (bots get renamed).

Takeaway

AI access is not one switch. It is sixteen crawlers, ten vendors, and a real split between training and citation. Blunt blocks feel tidy and quietly erase you from answers.

Rankify gives you labelled per-crawler control, robots.txt writing, plus the nearby AI/SEO toolkit (crawler log, referral tracking, llms.txt, schema, content analysis) when you are ready. Choose each vendor on purpose.

One Pro membership unlocks Rankify plus Progressify, Lightify, Generatify, and Snapshotify. New Pro plugins join existing memberships. Plans differ by site count only. See pricing when you want citation without handing every trainer a blank cheque.

Set the list once. Re-read robots.txt. When the labels match your policy, you stopped guessing.