AI Agency Crawler Policy Study 2026

We checked how AI agencies treat the crawlers that decide whether their own work is visible in AI search. The sample is 229 agency domains collected from GoodFirms, DesignRush and Clutch. For each one we read robots.txt, llms.txt and ai.txt on October 25, 2026.

224 agencies analysed successfully
1.8% block GPTBot in robots.txt
64.3% publish an llms.txt
92.4% allow all eight AI crawlers

What this means

Blocking is rare in this sample. 1.8% of the agencies block GPTBot, 0.9% block ClaudeBot and 0% block PerplexityBot; 92.4% do not block any of the eight crawlers we checked. That is the opposite of the popular claim that agencies are shutting AI crawlers out. The people who sell AI visibility have, by and large, left the door open.

The llms.txt number needs a caveat. 64.3% of these domains serve one (144 sites), but 15.3% of those files carry an SEO-plugin signature in the opening lines — Rank Math ships the feature and writes the file for you. We could only test for five plugin signatures, so the true share of automatically generated files is likely higher than 15.3%. Only 12.1% publish an ai.txt.

The quieter finding is that most agencies never address AI crawlers at all: 33.5% of robots.txt files name at least one AI agent. The rest rely on the wildcard, which is neither a policy nor an objection — just an absence of one.

Every domain in the sample

DomainGPTBotClaudeBotPerplexityBot Google-ExtendedCCBotllms.txtai.txt
24x7offshoring.com not named not named not named not named not named yes no
4soft.co not named not named not named not named not named no no
adaadigitom.com allowed allowed allowed allowed not named yes no
adastracorp.com not named not named not named not named not named yes no
agixtech.com allowed allowed allowed allowed blocked yes yes
aiautomationagency360.com not named not named not named not named not named yes no
aimtraction.com blocked not named not named not named not named no no
aisearchready.ca allowed allowed allowed allowed not named yes no
aisolutions.kz no robots.txt no robots.txt no robots.txt no robots.txt no robots.txt no no
aiveda.io allowed allowed allowed allowed not named yes no
ajackus.com not named not named not named not named not named no no
aktwins.com allowed allowed allowed allowed allowed yes no
aldanamotion.com not named not named not named not named not named no no
ancient.global not named not named not named not named not named yes no
anekalabs.com not named not named not named not named not named no yes
anml.com allowed allowed allowed allowed not named yes no
appinventiv.com not named not named not named not named not named yes no
appmakersla.com unavailable unavailable unavailable unavailable unavailable no no
appsters.io allowed allowed allowed allowed allowed no no
appsynergies.com not named not named not named not named not named no no
apptunix.com allowed allowed allowed not named allowed yes no
appverticals.com allowed allowed allowed allowed not named yes no
arbisoft.com not named not named not named not named not named yes no
arsenalitc.com not named not named not named not named not named yes no
azilen.com not named allowed not named allowed allowed yes no
azumo.com not named not named not named not named not named yes no
bchanel.com allowed allowed not named allowed allowed no no
bintime.com not named not named not named not named not named no no
bitcot.com allowed allowed allowed allowed allowed no yes
bitontree.com not named not named not named not named not named yes no
blink22.com not named not named not named not named not named no no
blubridge.com no robots.txt no robots.txt no robots.txt no robots.txt no robots.txt no no
brightn8.com unavailable unavailable unavailable unavailable unavailable no no
chopdawg.com not named not named not named not named not named no no
chudovo.com not named not named not named not named not named no no
codesclue.com allowed not named not named not named not named yes no
codingcops.com not named not named not named not named not named no no
coherentsolutions.com not named not named not named not named not named yes no
cropsly.com allowed allowed allowed allowed blocked yes no
cubix.co not named not named not named not named not named yes no
customerparadigm.com no robots.txt no robots.txt no robots.txt no robots.txt no robots.txt yes no
cybrisktech.com not named not named not named not named not named no no
data-science-ua.com not named not named not named not named not named no no
databyteworks.com not named not named not named not named not named yes no
dcrayon.com allowed allowed allowed allowed blocked yes no
debutinfotech.com allowed allowed allowed allowed not named yes no
designli.co allowed allowed allowed allowed allowed yes no
devarionsolution.com error:ssl
dianapps.com allowed allowed allowed allowed allowed yes no
diffco.us allowed allowed allowed allowed not named yes no
digitalpixora.com allowed allowed allowed allowed allowed yes yes
digitlhaus.com not named not named not named not named not named yes no
dreamztech.com not named not named not named not named not named yes yes
echoforgecloud.com blocked blocked not named blocked blocked no yes
edvantis.com not named not named not named not named not named no no
eleks.com not named not named not named not named not named yes no
empat.tech allowed allowed allowed allowed not named yes no
engineanalytics.tech not named not named not named not named not named no no
exatosoftware.com blocked blocked not named blocked blocked no no
exelanceit.in not named not named not named not named not named no no
exemplarymarketing.com not named not named not named not named not named no no
factoryjet.com allowed allowed allowed allowed allowed yes no
flyaps.com not named not named not named not named not named no no
focusreactive.com not named not named not named not named not named yes no
future-processing.com not named not named not named not named not named no no
geeks.ltd not named not named not named not named not named yes no
geekyants.com allowed allowed allowed allowed blocked yes no
genai-labs.io not named not named not named not named not named no no
gkncollective.com not named not named not named not named not named yes yes
globalnodes.com not named not named not named not named not named yes no
gloozd.agency no robots.txt no robots.txt no robots.txt no robots.txt no robots.txt no no
gojilabs.com not named not named not named not named not named yes no
goodface.agency not named not named not named not named not named yes no
goskyrocketai.com not named not named not named not named not named no no
granaura.ai allowed allowed allowed allowed not named yes no
greenmintlabs.com allowed allowed allowed allowed allowed yes no
growth-loop.io not named not named not named not named not named yes no
hashbyt.com not named not named not named not named not named no yes
helixstax.com allowed allowed allowed allowed allowed no no
helpware.com not named not named not named not named not named yes no
highpolar.io not named not named not named not named not named no yes
ibarts.in not named not named not named not named not named no no
ibrinfotech.com not named not named not named not named not named no no
ideamaker.agency allowed allowed allowed allowed allowed yes no
indatalabs.com not named not named not named not named not named yes no
indium.tech not named not named not named not named not named yes no
infograins.com not named not named not named not named not named no yes
infosense.tech not named not named not named not named not named yes yes
innovaciotech.com not named not named not named not named not named no no
inoxoft.com allowed allowed allowed allowed allowed yes no
instinctools.com not named not named not named not named not named yes yes
intersog.com allowed allowed allowed allowed allowed yes no
invergetech.com not named not named not named not named not named no no
inveritasoft.com not named not named not named not named not named no no
it-jim.com not named not named not named not named not named yes no
itransition.com not named not named not named not named not named yes no
jadeglobal.com not named not named not named not named not named yes no
junkiescoder.com not named not named not named not named not named yes yes
kanerika.com allowed allowed allowed allowed allowed yes yes
kevychsolutions.com not named not named not named not named not named yes yes
key4web.it not named not named not named not named not named yes no
kitrum.com not named not named not named not named not named no no
klearstack.com not named not named not named not named not named no no
kodexolabs.com not named not named not named not named not named yes no
krazimo.ai not named not named not named not named not named yes no
krazimo.com not named not named not named not named not named yes no
kreante.co not named not named not named not named not named yes no
krishaweb.com not named not named not named not named not named yes no
leanware.co allowed allowed allowed allowed not named yes no
leewayhertz.com not named not named not named not named not named no no
leobit.com allowed allowed allowed allowed allowed yes no
lewebsitetech.com unavailable unavailable unavailable unavailable unavailable no no
luby.co allowed allowed allowed allowed allowed yes no
lumitech.co not named not named not named not named not named yes no
mantravi.com error:ssl
markovate.com allowed allowed not named not named not named yes no
marutitech.com not named not named not named not named not named yes no
merixstudio.com not named not named not named not named not named no no
miirage.com not named not named not named not named not named no yes
mindrops.com allowed allowed allowed allowed allowed yes no
mobiloitte.us allowed allowed allowed allowed allowed yes no
mtechzilla.com not named not named not named not named not named yes no
mysticmediasoft.com not named not named not named not named not named no yes
naestinn.com not named not named not named not named not named yes no
ndovesha.ai not named not named not named not named not named no no
netsetsoftware.com not named not named not named not named not named yes no
neuramonks.com not named not named not named not named not named yes no
neurocell.in not named not named not named not named not named yes no
nextgensoft.io allowed allowed allowed not named not named yes no
nipsapp.com not named not named not named not named not named yes no
nix-united.com not named not named not named not named not named yes no
okaxi.com allowed allowed allowed allowed not named yes yes
ontiktechnology.com not named not named not named not named not named yes no
openxcell.com not named not named not named not named not named yes no
ortemtech.com allowed allowed allowed allowed allowed yes no
outloud.co not named not named not named not named not named no no
outsourcebigdata.com not named not named not named not named not named no no
oxagile.com not named not named not named not named not named yes no
pendoah.ai not named not named not named not named not named yes no
pharosproduction.com not named not named allowed not named not named yes no
phenomenonstudio.com allowed allowed allowed allowed allowed yes no
phenx.io not named not named not named not named not named yes no
pieoneers.com allowed allowed allowed allowed allowed yes no
pinnasys.com not named not named not named not named not named yes no
plavno.io allowed allowed allowed allowed allowed yes no
plego.com not named not named not named not named not named no no
pretobusiness.com not named not named not named not named not named no no
prismcrux.com not named not named not named not named not named no no
probeyservices.com not named not named not named not named not named no no
projectshub.co.in not named not named not named not named not named no no
promaticsindia.com not named not named not named not named not named yes no
quantumitinnovation.com not named not named not named not named not named no no
quytech.com allowed allowed allowed not named not named yes no
rebeldot.com not named not named not named not named not named no no
reluconsultancy.com allowed allowed allowed not named not named yes no
riseuplabs.com not named not named not named not named not named yes no
robotronix.co.in not named not named not named not named not named no yes
rockonesolutions.com allowed allowed allowed allowed not named no no
roonyx.tech not named not named not named not named not named no no
rootedai.co.in blocked allowed allowed blocked allowed yes no
saastoagent.com not named not named not named not named not named no no
samyotech.com not named not named not named not named not named yes no
scalosoft.com not named not named not named not named not named yes no
scopicsoftware.com not named not named not named not named not named yes no
sdlccorp.com not named not named not named not named not named yes yes
sdtcdigital.com not named not named not named not named not named no no
sfailabs.com not named not named not named not named not named no no
shamlatech.com allowed allowed allowed allowed allowed yes no
simform.com allowed allowed allowed allowed not named yes no
sitka.ai allowed allowed allowed allowed not named yes no
sketchdev.io not named not named not named not named not named yes no
slash.software not named not named not named not named not named yes no
sobot.io allowed allowed allowed allowed allowed yes no
softdoes.com not named not named not named not named not named yes no
solulab.com allowed allowed allowed allowed allowed yes no
sophi.link no robots.txt no robots.txt no robots.txt no robots.txt no robots.txt no no
spaculus.com allowed allowed allowed not named not named yes no
steer73.com not named not named not named not named not named no no
stssoftware.com not named not named not named not named not named yes no
stxnext.com not named not named not named not named blocked no no
successive.tech allowed allowed allowed allowed not named yes no
suffescom.com not named not named not named not named not named yes yes
sumatosoft.com allowed allowed allowed allowed allowed yes no
swacode.com not named not named not named not named not named no no
syljotech.com not named not named not named not named not named yes yes
takeaway-reality.com not named not named not named not named not named yes no
talentica.com not named not named not named not named not named yes no
team400.ai not named not named not named not named not named no yes
tech-stack.com not named not named not named not named not named yes no
tech2globe.com not named not named not named not named not named yes no
technoyuga.com not named not named not named not named not named yes no
techquadra.com allowed allowed allowed allowed not named yes no
techugo.com not named not named not named not named not named no no
techunting.net not named not named not named not named not named yes no
terrificminds.com not named not named not named not named not named no no
thedreamers.us not named not named not named not named not named yes no
thinktoshare.com not named not named not named not named not named no no
thirstysprout.com not named not named not named not named not named no no
triare.net not named not named not named not named not named yes no
trigma.com allowed allowed allowed allowed allowed yes no
uasoftdev.com allowed allowed allowed allowed allowed yes no
unicoconnect.com allowed allowed allowed allowed allowed yes no
unicsoft.com not named not named not named not named not named no no
upcoretech.com not named not named not named not named not named no no
useallfive.com not named not named not named not named not named yes no
utility.agency allowed allowed allowed allowed allowed yes no
uvik.net allowed allowed allowed allowed allowed yes no
valuecoders.com allowed allowed allowed not named allowed yes yes
vanie.ai not named not named not named not named not named no no
vectoriumlabs.com not named not named not named not named not named no no
vegavid.com allowed not named allowed allowed allowed yes no
velorabyte.com not named not named not named not named not named no no
visionify.ai not named not named not named not named not named no no
wavect.io allowed allowed allowed allowed allowed yes no
wearebrain.com not named not named not named not named not named yes no
wearenotch.com not named not named not named not named not named yes no
webisoft.com allowed allowed allowed allowed allowed yes no
websem.ro allowed allowed allowed allowed allowed yes no
whamply.com not named not named not named not named not named no no
whizzbridge.com not named not named not named not named not named yes no
wildnetedge.com not named not named not named not named not named yes no
wiserbrand.com not named not named not named not named not named no no
wislacode.com allowed allowed allowed allowed not named yes no
yalantis.com not named not named not named not named not named no no
yotomations.com not named not named not named not named not named yes no
zerotasklabs.com not named not named not named not named not named no no
ziplyne.agency allowed allowed allowed allowed allowed yes yes
zipnom.com not named not named not named not named not named yes yes
zone3000.com not named not named not named not named not named no yes
Download raw data (JSON) Read methodology

Collected October 25, 2026. 2 domains could not be analysed (DNS, TLS, timeouts, bot protection) and are excluded from the percentages.