# Methodology — AI Agency Crawler Policy Study 2026

## What was measured

For each domain in the sample we requested three files over HTTPS and recorded the raw result:

1. `/robots.txt` — parsed for per-user-agent rules
2. `/llms.txt` — HTTP status plus the first 1,000 characters of the body
3. `/ai.txt` — HTTP status only

## Sample

| Source | URL | Pages used | Domains contributed |
|--------|-----|------------|---------------------|
| GoodFirms | https://www.goodfirms.co/artificial-intelligence | pages 1-4 | 192 |
| DesignRush | https://www.designrush.com/agency/ai-companies | pages 1-2 | 38 |
| Clutch | https://clutch.co/developers/artificial-intelligence | page 1 | 9 |

Domains were extracted from the outbound agency links on each listing page, reduced to their registrable host
(scheme, `www.`, path and query string removed), then de-duplicated. `229` unique domains were queued;
**224** were analysed successfully and are included in the percentages below.

Domains that could not be reached are excluded from every percentage and reported separately.

## Collection period

Single snapshot, collected **2026-10-25**. `robots.txt` and `llms.txt` change often; these figures describe one day,
not a stable property of any site.

## How "allowed" and "blocked" were determined

`robots.txt` was parsed into user-agent groups, without using a library, so that an agent that is not named can be
distinguished from one that is named and allowed.

For each of `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot`, `anthropic-ai`, `ChatGPT-User` and
`Bytespider`, the status recorded is one of:

| Value | Meaning |
|-------|---------|
| `blocked` | The agent has its own group and the rule set disallows `/` |
| `allowed` | The agent has its own group and `/` is permitted |
| `not_mentioned` | The agent has no group of its own; the wildcard `User-agent: *` group applies |
| `no_robots_txt` | The domain returned 404 for `/robots.txt`, so nothing is restricted |
| `unavailable` | `robots.txt` returned a non-404 error status |

Matching follows the longest-path rule: when several `Allow` and `Disallow` rules match a path, the most specific
one wins, and `Allow` wins a tie. A `Disallow: /` inside a named group is what we count as a block.

A domain returning 200 for `llms.txt` is only counted as having one if the response carried a non-empty, non-HTML
body. Some hosts answer unknown paths with a 200 HTML page; `10` domains in this sample did exactly that and were
not counted as having the file.

## Limitations

- **The sample is not random.** It is whoever the three directories list on the pages we read. Agencies that do not
  buy directory placement are absent.
- **Some domains could not be reached.** `2` of `229` failed with:

| Error type | Domains |
|------------|---------|
| `ssl` | 2 |

  Most are TLS failures, DNS failures or bot protection (Cloudflare and similar) returning 403/429 to a non-browser
  client. A site that blocks our request is not necessarily blocking AI crawlers; those results are simply unknown.
- **Redirects were followed.** The file checked is the one served at the final URL, which may be on another host
  (for example a `www.` variant). The original domain is what appears in the table.
- **No JavaScript rendering.** Only server-delivered content was read.
- **`llms.txt` count includes plugin output.** Many files are auto-generated by SEO plugins rather than written
  deliberately. We tested the opening 1,000 characters for five signatures (Rank Math, Yoast, All in One SEO,
  AIOSEO, SEOPress); `22` of `144` files matched.
  Because only five signatures were tested, the true share of automatically generated files is likely higher than
  the 15.3% measured.
- **A single snapshot.** No historical comparison.

## Reproducing the study

The collector and analyser are in `study/`:

```bash
cd study
python3 collect.py     # reads data/domains.txt, writes data/results.json incrementally
python3 analyze.py     # writes data/summary.json
```

Requirements: Python 3 and `requests`. The collector sends `Web2AI-Research-Bot/1.0 (+https://web2ai.eu/study/)`,
waits one second between domains, and uses 10 s / 10 s / 5 s timeouts for `robots.txt`, `llms.txt` and `ai.txt`.
Dates and percentages change as the underlying files change; re-run to refresh.