What Is llms.txt?
llms.txt is a plain-text policy file at the root of your domain — like /robots.txt, but for AI crawlers: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, anthropic-ai, CCBot, and the new wave of agentic LLM agents. It tells them which paths they can read, which to skip, and which pages matter most.
This generator builds a valid file in seconds with sensible defaults, auto-detects your real paths from /sitemap.xml, and lets you override any path's priority manually.
How to Use the Generator
- Enter your website URL.
- Optionally list your top pages (one per line) — they'll be marked high priority.
- Optionally list paths to ignore (admin, staging, internal docs) — they'll be added to the Disallow list.
- Toggle Allow AI crawlers based on your policy. Off = explicit Disallow rules for every major AI bot.
- Click Generate, then Copy or Download .txt.
- Upload the file to
https://yoursite.com/llms.txt.
How Priority Is Auto-Classified
- High:
/blog/,/docs/,/guides/,/tutorials/, and the homepage. - Medium:
/tools/,/products/,/features/,/pages/. - Low:
/tags/,/archives/,/search, author archives, paginated lists. - Disallow:
/admin/,/api/,/auth/,/dashboard/,/account/,/wp-admin/.
Frequently Asked Questions
+What is llms.txt?
llms.txt is a plain-text file at the root of your domain that tells AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, etc. — which parts of your site they can read, which to skip, and which pages matter most.
+How do I install the generated file?
Save the generated text as llms.txtand upload it to the root of your website so it's reachable at https://yourdomain.com/llms.txt — same place you put robots.txt.
+Does Google or OpenAI actually read llms.txt?
The standard is emerging. Google-Extended, GPTBot, ClaudeBot, and several agentic crawlers already respect User-agent / Disallow directives in llms.txt-style files. Priority annotations are non-standard but increasingly parsed by retrieval-style LLM agents.
+Should I block AI crawlers?
Depends on your business. Most content sites benefit from being included — AI overviews drive referral traffic and brand awareness. Subscription, premium-content, and unique-research sites often opt out.
+Why are admin, dashboard, and API paths disallowed by default?
Those paths typically expose private data, authenticated user areas, or backend endpoints that should never appear in AI training data or search results.
+Is this tool free?
Yes. The tool is 100% free with no signup, no credit card, and no email required. Generate as many llms.txt files as you want.