Somewhere on your site, a file most people never open is quietly deciding who gets in. That file is robots.txt, and it’s been around since 1994, built for search engines, not AI. A newer file called llms.txt showed up in 2024, promising something bigger: a curated map of your best content, built specifically for AI models.
One of these files actually controls access. The other is a request that most AI companies still haven’t agreed to honor. This guide covers both, through Generative Engine Optimization, what each one actually does, and how to set them up correctly.
RIja Ghayas
·
September 18, 2026
·
11 min read
Quick answer
llms.txt gives AI systems a curated summary of your site. It’s newer, optional, and far less consistently used than people assume.
→
Sort your AI crawlers into two groups. Training crawlers feed a model’s future knowledge. Search crawlers power what a model cites right now. Blocking one doesn’t affect the other.
→
Decide what you actually want to allow. Some brands want AI citations but not AI training. Some want neither. There’s no universal right answer, only a decision you should make on purpose.
→
Write your robots.txt rules explicitly. Don’t rely on defaults. A lot of hosting platforms block AI crawlers automatically, without telling you.
→
Add llms.txt if it fits your site. It won’t boost your rankings. It can genuinely help AI tools understand a large, complex site faster.
How It Works
01
Training Crawlers Build a Model’s Future Knowledge
GPTBot, ClaudeBot, and similar bots fetch content in bulk, without urgency, to feed a future training run. Block one of these, and your content simply never enters that model’s training data. It has no effect on today’s AI answers.
02
Search and Agent Crawlers Power Live Answers
OAI-SearchBot, Claude-SearchBot, and PerplexityBot fetch content in something closer to real time, to answer questions being asked right now. Block one of these, and you can disappear from that platform’s AI answers immediately.
03
robots.txt Is a Request, Not a Lock
It’s formalized as an actual internet standard, RFC 9309, but it only works because crawlers choose to follow it. Most major AI labs say they comply. Not all of them actually do.
Comparison
robots.txt
llms.txt
Compliance
Mandatory in practice, every reputable crawler checks it
Compliance
Optional, a courtesy some AI systems extend
Function
Blocks or allows, a binary switch per bot, per path
Function
Curates, points AI systems toward your best content
Origin
Built for search engines in 1994, formalized as RFC 9309
Origin
Proposed by Jeremy Howard of Answer.AI in September 2024
Official status
An accepted internet standard
Official status
A community convention, not an official standard
Google has stated directly that it doesn’t use llms.txt for ranking or for its AI features. OpenAI and Anthropic haven’t made the same blanket commitment either, though both show up reading it in some contexts.
Context
Two things are pushing this decision onto every site owner’s desk right now, whether they’re ready or not.
01
Cloudflare Is Changing Its Default AI Bot Settings
Cloudflare is changing its default AI bot settings on September 15, 2026, but not for every site. The new defaults apply to new domains onboarding to Cloudflare and to existing free-tier accounts that haven’t set an explicit preference, and only on pages that show ads. If you’re in a configured paid zone, nothing changes automatically, but it’s still worth confirming your settings say what you think they say.
02
Perplexity Has Been Caught Running Undeclared Crawlers
Cloudflare published evidence that Perplexity used crawlers outside its declared user-agents, rotating identities to get around robots.txt blocks. Perplexity has also stated that its user-triggered fetcher, Perplexity-User, counts as an agent rather than a bot, and therefore isn’t bound by robots.txt at all. That stance has caused real disputes with publishers.
Metrics
Every AI crawler falls into one of these categories. Decide deliberately, rather than defaulting to whatever your hosting platform shipped with.
RELATED GUIDES
Framework
Here’s a working robots.txt block that allows AI search and agent crawlers while blocking training crawlers, a common middle-ground setup.
Test Every Change in Staging First
A single misplaced Disallow can silently remove you from AI answers with no warning, no error message, nothing.
Set a Quarterly Review Reminder
New crawlers launch regularly, and existing ones occasionally rename themselves. A rule you set once can quietly go stale.
Keep llms.txt Genuinely Curated
If you’re adding one, the whole point is a short, useful map of your best content. A dumped copy of your full sitemap defeats the purpose entirely. Only the H1 is technically required. A one-line summary and a handful of links to your best pages covers most sites well.
robots.txt
// Block training crawlers
// Allow search & agent crawlers
How It Works
01
Treat Your Rules as Living Configuration, Not a One-Time Setup
New AI crawlers launch several times a year, and existing ones occasionally rename themselves or split into new bots, the way OpenAI and Anthropic have both done. A robots.txt file you wrote once and forgot about drifts out of date quietly.
02
Re-Check Your Rules Every Time You Change Infrastructure
A new CDN, security plugin, or hosting migration can silently overwrite your robots.txt or add its own AI-blocking defaults. Treat any infrastructure change as a trigger to verify your crawler rules still say what you think they say.
03
Follow Each AI Lab’s Own Documentation, Not Just Practitioner Roundups
GPTBot, ClaudeBot, and PerplexityBot’s actual behavior is defined by their operators, not by blog posts about them. Anthropic, OpenAI, and Perplexity all publish and periodically update their own crawler documentation, that’s the source to trust when it conflicts with secondhand advice.
04
Revisit Your Allow/Block Decisions as Your AI Visibility Goals Change
A blanket block made in 2024 out of caution may no longer match what you actually want in 2026. Review your training vs. search crawler decisions against your current AI visibility goals, not the assumptions you made when you first set them.
Debunked
•
llms.txt will boost your AI search rankings. It won’t, by design. Jeremy Howard, the person who proposed it, built it to help coding tools and AI agents parse documentation faster. He never pitched it as a visibility or citation tool.
•
Google reads and uses llms.txt for AI Overviews. It doesn’t. Google has said directly that Search, including its AI features, doesn’t use llms.txt at all.
•
Every major AI company officially supports llms.txt. Support is genuinely mixed. Anthropic and Perplexity show the most consistent engagement with it. OpenAI’s behavior is less clear and unconfirmed publicly. Treat any claim of universal support with real skepticism.
•
Blocking a crawler in robots.txt stops all of that company’s access. Not always. Some companies split training and search into entirely separate crawlers with separate user-agents. Blocking one doesn’t touch the other.
Add llms.txt because it’s low-cost and genuinely useful for a complex site, not because someone promised it would move your rankings.
Tools
Check Your Server Logs Directly
Look for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and similar strings in your access logs. This shows you which crawlers are actually visiting, not just which ones you’ve configured rules for.
Verify by IP Range Where It Matters
Real GPTBot and Googlebot traffic can be confirmed against published IP ranges. Anyone can send a request claiming to be GPTBot. Only an IP check actually proves it.
Use Google’s Rich Results Test and Search Console
These confirm your site is reachable and correctly structured for Google’s own crawlers, separate from the AI-crawler question entirely.
Track the Aftermath
Re-run this check quarterly. New crawlers launch regularly enough that a one-time audit goes stale.
Watch for
Blocking AI Crawlers by Accident
A security plugin or a CDN default can block AI bots without anyone deciding to. Check your actual live robots.txt, not the one you think you configured.
Confusing Google-Extended With Googlebot
Blocking Google-Extended only opts you out of Gemini training. It does not touch your regular Google Search rankings.
Trusting Every User-Agent String at Face Value
Spoofed traffic exists. If a bot's behavior looks off, verify the IP before you trust the name.
Treating llms.txt as a Ranking Lever
It’s curation, not citation currency. Set your expectations accordingly, and you won’t be disappointed when it doesn’t move a single ranking.
Checklist
✓
Confirm your live robots.txt matches what you intended. Check the actual file at yoursite.com/robots.txt, not the config you think you saved, hosting platforms and security plugins can silently override it.
✓
Check your server logs for crawler activity. Look for GPTBot, ClaudeBot, PerplexityBot, and similar strings to see who’s actually visiting, not just who you’ve written rules for.
✓
Verify traffic against published IP ranges for anything that matters. A user-agent string alone can be spoofed by anyone.
✓
Track AI citations and mentions after any change. A robots.txt edit should show up in your AI visibility data within a few weeks if it’s working as intended.
✓
Re-run this check quarterly. New crawlers launch regularly enough that a one-time audit goes stale.
Author
Related Articles
Chat GPT
How to Optimize Your Website for ChatGPT
Learn how to optimize your site for ChatGPT...
GEO vs SEO
What's the Difference Between GEO and SEO?
SEO and GEO both improve online visibility...
