Is llms.txt an official standard?

No. It's an open proposal from 2024 that some sites and tools have adopted. There's no standards body behind it and no requirement to use it.

Will llms.txt get me recommended by ChatGPT?

Not on its own. No major assistant has said it uses llms.txt as a signal. Clear pages, structured data, consistent details and presence on trusted sources matter far more.

Should I block AI crawlers?

If you want to be recommended in AI answers, don't block the search crawlers such as OAI-SearchBot, ClaudeBot and PerplexityBot. Blocking training crawlers is a separate choice.

llms.txt Explained: What It Is and Whether You Need One

Short answer

llms.txt is a short Markdown file at the root of your website (yoursite.com/llms.txt) that summarises who you are and links to your most useful pages, written for AI tools rather than people. It was proposed in 2024 as an open convention. It's cheap to add and harmless, but no major AI company has said it relies on it, so treat it as a small extra, not a strategy.

What llms.txt is

The idea, proposed by Jeremy Howard of Answer.AI in September 2024, is simple: websites are built for people and full of navigation, scripts and layout. A language model trying to understand a site benefits from a short, clean summary that points to the pages that matter. llms.txt is that summary.

The proposed format is plain Markdown: a top-level heading with your name, a short summary in a blockquote, optional paragraphs of detail, then sections of links, each with a one-line note. A section called "Optional" marks links a tool can skip if it's short on space. Some sites also publish an llms-full.txt with the full text of key pages, a convention that grew up around documentation sites.

Who actually reads it

This is where honesty matters. As of late 2026, none of the big assistant providers has said that its search or answers use llms.txt as a signal, and Google representatives have said Google Search doesn't use it. Some AI tools, coding assistants and agents do fetch it when they're pointed at a site.

So why add one? Because it takes an hour, it can't hurt, and writing it forces you to state plainly what you are and which pages answer the important questions. That clarity is exactly what helps you on the pages assistants do read.

An example for a hotel

Here's a complete llms.txt for a fictional hotel. Keep yours this short.

# Harbour Lane Hotel

> A 4-star, 84-room boutique hotel in Dubai Marina, a 5-minute walk from JBR beach,
> with a rooftop pool, family rooms and free parking. Direct bookings at
> https://www.harbourlanehotel.example/book.

Check-in from 15:00, check-out by 12:00. Pets are not allowed. Airport transfers on request.

## Key pages

- [Rooms and suites](https://www.harbourlanehotel.example/rooms): room types, sizes and views
- [Families](https://www.harbourlanehotel.example/families): family rooms, kids' club, cots
- [Rooftop pool and brunch](https://www.harbourlanehotel.example/rooftop): hours and booking
- [Location and parking](https://www.harbourlanehotel.example/location): directions, parking, transfers
- [FAQ](https://www.harbourlanehotel.example/faq): policies and practical questions

## Optional

- [Press](https://www.harbourlanehotel.example/press)

How to write a good one

Lead with facts: type of business, location, size and the two or three things that set you apart.
Link only to pages that answer questions: rooms, menus, families, location, policies, FAQ. Skip pages that exist for decoration.
Keep it current. Old prices or a closed restaurant in llms.txt undo the trust you're building.
Match your site and listings: same name, same address, same phone.
Serve it as plain text at exactly /llms.txt, without a login or a cookie wall.

llms.txt isn't robots.txt

llms.txt doesn't control access; it's a summary. Whether AI crawlers can read your site at all is decided by robots.txt, and by any firewall or security plugin in front of your site. The crawlers worth knowing include OpenAI's OAI-SearchBot (search) and GPTBot (training), Anthropic's ClaudeBot, and PerplexityBot. Google's Google-Extended token lets you control use of your content for Gemini models without affecting Google Search.

If you want to be recommended, make sure the search crawlers aren't blocked. A simple robots.txt that welcomes them looks like this:

# Search engines and AI search crawlers: welcome
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://www.harbourlanehotel.example/sitemap.xml

Whether to allow training crawlers such as GPTBot is a business decision. Blocking them doesn't stop the separate search crawlers from finding you.

How to check yours

Open yoursite.com/llms.txt in a browser: it should show plain text, not your homepage or a 404.
Open yoursite.com/robots.txt and look for Disallow: / under any AI crawler you want to allow.
Ask your host or security provider whether "AI bot blocking" is switched on. Several content-delivery networks now offer it, sometimes on by default.

llms.txt explained: what it is and whether you need one