What Is llms.txt?
llms.txt is a plain-text policy file at the root of your domain — like /robots.txt, but for AI crawlers: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, anthropic-ai, CCBot, and the new wave of agentic LLM agents. It's an emerging convention meant to signal which paths matter most and which to skip — but it is not yet universally adopted, and major AI crawlers don't currently parse a separate /llms.txt file for access control (they honor robots.txt for that).
This generator builds a valid file in seconds with sensible defaults, auto-detects your real paths from /sitemap.xml, and lets you override any path's priority manually.
The llms.txt proposal was published in September 2024 by Jeremy Howard (co-founder of Answer.AI, fast.ai, and Kaggle), with the goal of helping Large Language Models (LLMs) understand a site's structure and key content in a way that's far more compact than parsing the whole HTML tree. The format is intentionally Markdown-based — an H1 for the site name, a quote block for the tagline, top-level H2 sections for the major content categories, and bullet links to individual important pages with short descriptions. Because it's Markdown, AI models can read it efficiently without spending tokens on navigation, boilerplate, or ads.
There's also a llms-full.txt variant that includes the complete content of every linked page concatenated into one file — meant as a ready-to-feed corpus for AI fine-tuning, content summarisation, or retrieval-augmented generation (RAG) pipelines. Sites publishing both files signal that they welcome being used as a knowledge source. A clear distinction worth understanding: robots.txt tells crawlers what they may or may not fetch (access control); llms.txttells LLMs how to make sense of the content they do fetch (interpretation guidance). The two files serve completely different purposes and should co-exist on the same domain. llms.txt won't block AI training — for that you need robots.txt directives targeting AI-specific user agents like GPTBot, ClaudeBot, anthropic-ai, CCBot, and Google-Extended.
Adoption is in its early stages but accelerating quickly. Major developer-tool and documentation sites — Vercel, Mintlify, Anthropic's own docs — have already shipped llms.txt files. Search engines and AI assistants haven't formally announced support, but Anthropic and OpenAI have both indicated interest in structured site-guidance files, and there's an active spec.llmstxt.org community refining the standard. The cost of publishing one is minutes; the upside if the standard matures is meaningful representation in AI-mediated search and assistant answers.
How to Use the Generator
- Enter your website URL.
- Optionally list your top pages (one per line) — they'll be marked high priority.
- Optionally list paths to ignore (admin, staging, internal docs) — they'll be added to the Disallow list.
- Toggle Allow AI crawlers based on your policy. Off = explicit Disallow rules for every major AI bot.
- Click Generate, then Copy or Download .txt.
- Upload the file to
https://yoursite.com/llms.txt.
Common Use Cases
- Brand-mention accuracy in AI responses — if ChatGPT, Claude, or Perplexity ever cite your company, llms.txt gives them the canonical name, description, and key pages to reference. Without it, AI models lean on whatever snippets they happened to index, which can be stale, partial, or wrong.
- Knowledge-base and documentation sites — feed structured info to AI assistants that increasingly answer technical questions. A well-curated llms.txt helps assistants surface your documentation when users ask about your tool, library, or API.
- Product documentation and API references — when developers ask Claude or ChatGPT "how do I integrate with Stripe?", the answer quality depends on what the model can find. Publishing llms.txt with your API endpoints, authentication flow, and example code surfaces them more reliably.
- Open-source project visibility — library maintainers can use llms.txt to summarise the project, point to the README, link the API reference, and describe common usage patterns — making AI tooling more accurate when users ask about your library.
- News and journalism content control — publishers can guide how AI tools summarise their content, including attribution requirements, fact-checking pointers, and editorial-style notes.
- Personal portfolio and professional bio — freelancers and consultants can ensure AI represents their professional background accurately rather than mixing up people with similar names or surfacing outdated bios from old profiles.
How Priority Is Auto-Classified
- High:
/blog/,/docs/,/guides/,/tutorials/, and the homepage. - Medium:
/tools/,/products/,/features/,/pages/. - Low:
/tags/,/archives/,/search, author archives, paginated lists. - Disallow:
/admin/,/api/,/auth/,/dashboard/,/account/,/wp-admin/.
llms.txt Generator vs Other Tools
Versus writing llms.txt manually — the spec is still emerging and format details matter. Tooldit ensures syntactically correct output with auto-classified priorities.
Versus skipping llms.txt entirely — some sites argue it's not worth the effort yet (low adoption). But early adoption costs nothing and may influence how AI assistants surface your content as the standard matures.
Versus robots.txt — robots.txt blocks crawlers; llms.txt invites AI to use specific pages. They serve opposite roles — both can coexist on the same site.
Versus sitemap.xml — sitemaps list every URL for search engines. llms.txt curates a subset of high-value pages for AI consumption with priority hints.
Troubleshooting & Common Issues
- Priority classification wrong — the auto-classifier guesses from URL patterns and content type. Edit the output manually to fix — the priority hint is human-readable text, not a strict format.
- AI assistants ignore my llms.txt — adoption is still partial as of 2025. ChatGPT, Claude, and Perplexity are starting to honor it; others haven't implemented it yet. Don't expect immediate effect.
- File placement — llms.txt goes at your domain root (https://example.com/llms.txt), same as robots.txt and sitemap.xml.
- Sensitive pages included by mistake — the generator includes whatever you point it at. Review the output for paywalled, private, or admin URLs and remove them before deploying.
- Format syntax errors — the llms.txt spec is markdown- based but with specific section headers. Tooldit outputs the standard format; if you edit by hand, check spec.llmstxt.org for current syntax.
Frequently Asked Questions
+What is llms.txt?
llms.txt is a plain-text file at the root of your domain that tells AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc. — which parts of your site they can read, which to skip, and which pages matter most.
+How do I install the generated file?
Save the generated text as llms.txtand upload it to the root of your website so it's reachable at https://yourdomain.com/llms.txt — same place you put robots.txt.
+Does Google or OpenAI actually read llms.txt?
Not as a standalone access-control file — at least not yet. llms.txt is an emerging convention, not a universally adopted standard. Crawlers like GPTBot, ClaudeBot, and Google-Extended respect User-agent / Disallow directives in robots.txt, but support for reading a separate /llms.txtfile isn't established, and major AI crawlers don't currently parse it. Publishing one is a forward-looking signal, so keep your real access rules in robots.txt too.
+Should I block AI crawlers?
Depends on your business. Most content sites benefit from being included — AI overviews drive referral traffic and brand awareness. Subscription, premium-content, and unique-research sites often opt out.
+Why are admin, dashboard, and API paths disallowed by default?
Those paths typically expose private data, authenticated user areas, or backend endpoints that should never appear in AI training data or search results.
+Is the llms.txt generator free to use?
Yes. Generating and downloading llms.txt files is 100% free — no signup, no credit card, and no email required. Build as many llms.txt files as you want, with unlimited paths and priority rules.
+Will llms.txt replace robots.txt?
No — they serve different purposes. robots.txt controls which crawlers can access content; llms.txt describes which content is best for AI to use. Most sites should have both.
+Is llms.txt an official standard?
It's a community proposal (spec.llmstxt.org by Jeremy Howard) that's gaining traction as AI assistants look for structured content guides. Not yet a W3C standard but increasingly recognised.