Why Most AI Agencies Don't Have an AI Crawler Policy

An agency that sells AI search visibility has one obvious professional obligation: decide what AI crawlers are allowed to do with its own site. We checked whether they have made that decision. Across 229 agency domains, the answer is mostly no.

Only 33.5% of the robots.txt files in our sample mention a single AI crawler by name. The rest rely on the wildcard, which is not a policy — it is the absence of one. That distinction matters more than the blocking numbers, because it tells you what these companies have actually thought about.

This article is about what we found, why the gap exists, and what an agency selling AI visibility should do about it before the end of the year. The full dataset and method are in the AI Agency Crawler Policy Study 2026.

What we measured

We collected 229 domains of agencies that market AI or machine learning services, drawn from three directories: GoodFirms, DesignRush and Clutch. Of those, 224 produced a usable answer; five were excluded — two failed at the TLS layer, three returned an unusable robots.txt.

For each domain we fetched three files: robots.txt, llms.txt and ai.txt. We parsed robots.txt manually rather than with a library, because the popular parsers collapse a critical distinction: an agent that is explicitly allowed is not the same as an agent that was never mentioned. A wildcard Disallow: / applies to everyone; an explicit Allow: / for GPTBot is a decision someone wrote down.

The full methodology, including the exact matching rules and the list of limitations, is published at study/data/methodology.md. The raw results are downloadable as JSON from the study page.

What we found

Blocking, it turns out, is rare. Out of 224 analysed domains:

• 1.8% block GPTBot (4 domains)
• 0.9% block ClaudeBot (2)
• 0.0% block PerplexityBot (0)
• 1.3% block Google-Extended (3)
• 3.1% block CCBot (7)

CCBot is the most-blocked crawler in the sample, and that is the one result that makes immediate sense. CCBot belongs to Common Crawl, a non-profit that publishes an open corpus used to train models. Blocking it is a data-licensing position, not a visibility decision. The crawlers that actually answer user questions — GPTBot, ClaudeBot, PerplexityBot — are blocked by almost nobody.

The aggregate confirms it: 92.4% of the sample block none of the eight crawlers we checked. Not one domain blocks all eight.

Then there is the number that surprised us. Only 33.5% of robots.txt files name at least one AI crawler explicitly. Nearly two thirds of these agencies have a file that says nothing at all about the systems their business depends on.

What the numbers do not say

Three cautions before anyone quotes these figures.

The sample is not random. It is whoever GoodFirms, DesignRush and Clutch list on the pages we read — 84% of it comes from GoodFirms alone. Agencies that buy directory placement are not the same population as agencies that do not. A larger, stratified sample could move these percentages by several points in either direction.

Silence is ambiguous. When a robots.txt does not name GPTBot, we record "not mentioned" — but that state covers two very different companies: one that deliberately allows AI crawlers and one that has never opened the file. Nothing in the data distinguishes them. That ambiguity is the finding, not a flaw in it, but it should stop anyone from reading silence as approval.

This is a snapshot, not a trend. Everything was collected on 25 October 2026. robots.txt files change after incidents, migrations and policy announcements. One of the sites in our sample may have rewritten its file the day after we looked. Repeating the crawl quarterly is the only way to know whether the gap is closing.

Why most agencies have no policy

It would be easy to read that as negligence. It is more accurate to read it as a structural gap, and there are four reasons it happens.

1. The file was written before AI crawlers existed

Most robots.txt files on the web were created once, by a hosting provider, a developer at launch, or a CMS plugin, and never touched again. A file written in 2019 could not have an opinion about GPTBot, which was announced in 2023. Silence is not a stance; it is a timestamp.

2. Nobody owns crawl policy

An agency's own website sits in an awkward organisational place. It is a marketing asset, maintained by whoever has bandwidth, usually measured on leads rather than on technical hygiene. Crawl policy sits between marketing, engineering and legal — and therefore belongs to no one. Client sites get a crawl audit because the contract says so. The agency's own site gets whatever the last person left behind.

3. The default is permissive, so nothing breaks

A missing policy does not produce an error. Pages get indexed, traffic arrives, the site looks fine. There is no feedback loop that punishes silence, which is exactly why silence persists. The cost is invisible until it isn't.

4. Plugins create an illusion of policy

This is the newest and most misleading factor. Several SEO plugins now generate an llms.txt automatically. In our sample, 64.3% of domains serve an llms.txt, but at least 15.3% of those files carry an SEO-plugin signature in the opening lines — Rank Math writes one for you. A file appears. Nobody decided anything. The dashboard shows a green tick for a policy that does not exist.

The same pattern shows up on the crawler side: a plugin or host may write a permissive robots.txt that happens to allow AI crawlers. The outcome looks identical to a deliberate decision, and the difference only surfaces when the decision needs to change.

What "no policy" actually costs

Two failure modes, in opposite directions.

The invisible opt-out. Someone blocks AI crawlers — often by copying a robots.txt snippet from a blog post written during the 2023 scraping panic — and forgets. The site keeps selling AI visibility while being unreadable to the systems that produce it. In our sample this affected a handful of domains, but the mechanism matters more than the count: the decision was made once, by someone who has since left, and never revisited.

The unexamined default. The site allows everything without ever deciding what that means for training data, licensing, or attribution. That is a defensible position, but only if it is a position. When a client asks "what is your policy on AI crawlers", the answer "the wildcard handles it" is not an answer.

There is a third, subtler cost: credibility. An agency that publishes a crawler study and then has an unexamined robots.txt is one search away from being embarrassed. We wrote this article partly because we audited our own site first.

What an AI agency should do in 2026

The fix is small. It takes under an hour and it should be documented, not just implemented.

Step 1 — Read your own robots.txt

Open https://yourdomain.com/robots.txt in a browser. You are looking for three things: whether it exists at all, whether there is a wildcard Disallow: /, and whether any AI user-agent appears by name. If it returns a 404, nothing is restricted — including crawlers you might not want.

Step 2 — Decide per crawler, not globally

The crawlers are not interchangeable. GPTBot and Google-Extended collect training data. ChatGPT-User and PerplexityBot typically fetch a page because a user asked about it — blocking those removes you from the answer, not just from the corpus. ClaudeBot and anthropic-ai play a similar dual role. CCBot feeds a public dataset.

A practical default for an agency selling AI visibility: allow the retrieval agents, think separately about training, and write the reasoning down.

The distinction is worth spelling out, because the names do not tell you which is which. GPTBot crawls to build training data; blocking it removes you from the corpus but does not remove you from an answer that the model already knows how to give. ChatGPT-User fetches a page because a person asked ChatGPT about it; blocking that one removes you from the conversation happening today. CCBot feeds Common Crawl, a public dataset that many models train on — a licensing question dressed as a technical one. Google-Extended is a training control, not a search control: blocking it does not affect your Google rankings.

An agency can reasonably arrive at different answers for each of those four. What it cannot reasonably do is have no answer for any of them.

Step 3 — Write the names down explicitly

Even a permissive rule should be explicit. This costs nothing and converts an absence into a decision:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

The operational benefit is real: the next person to touch the file can see what was intended. The analytical benefit is that your own site stops being an exception in audits like this one.

Step 4 — Treat llms.txt as hygiene, not leverage

Publish one if you want, but do not expect it to move citations. Google has said it does not use the file, and there is no published evidence of a citation uplift. What it does buy you is a clean summary for agents and a forcing function to decide what your most important pages are. If you publish one, write it by hand and keep it current — a stale llms.txt is worse than none.

Step 5 — Record the decision

A line in your internal docs: date, who decided, which crawlers, why. Policies that live only in a file get overwritten by the next migration. Policies with a rationale survive.

How to check your own site in ten minutes

Three commands and one judgement call:

• curl -s https://yourdomain.com/robots.txt — read it. Look for AI user-agents.
• curl -sI https://yourdomain.com/llms.txt — 200 means something is there; check whether it is a real text file or an HTML page returned with a 200 status (we found 10 of those in this sample).
• curl -sI https://yourdomain.com/ai.txt — 12.1% of the sample has one; most do not.

The judgement call is the actual work: decide whether the current state is what you want, and whether you can defend it to a client. If a prospect asks what your crawler policy is, the answer should fit in a sentence.

What this means for your own visibility

The larger lesson from the dataset is that AI visibility is still mostly an unclaimed field. The agencies selling it have, by and large, not configured their own sites for it. That is an opportunity, and it is also a warning: the advantage belongs to whoever treats the basics as basics.

It is worth being precise about what "configured" means here, because a permissive default is not the same as readiness. A site can allow every crawler and still be a poor source: thin pages, no entity markup, no author attribution, no structured data describing the business. Access is necessary and nowhere near sufficient. The 92.4% of domains that block nothing have opened the door; most have not yet furnished the room.

The reverse is also true, and it is the failure mode we would flag first. An agency that blocks the retrieval crawlers has closed the door and kept the sign. That is a strategic error that no amount of content work can compensate for, and it is invisible in every dashboard that measures traffic rather than access.

Crawl policy is the floor, not the ceiling. The work that moves a brand into AI answers is still structured content, entity clarity, and being worth citing — the same foundations we cover in our AEO and GEO practice, and the technical groundwork we implement through AI-ready web development. But it starts with a file that says what you actually want.

Conclusion

229 agencies. 224 usable results. 33.5% naming a crawler. 1.8% blocking GPTBot. Those numbers describe a market that has not yet decided what it thinks about the technology it sells. That is not a scandal — it is a gap, and gaps close.

The agencies that write their policy down this year will be the ones who can answer the question credibly next year. If you want to see the full breakdown, every domain is in the study, with the raw JSON published alongside it.

Sources: OpenAI: overview of GPTBot; Anthropic: how site owners can block ClaudeBot; llmstxt.org: the llms.txt proposal.