amisearchableami.cc

Guides

llms.txt vs robots.txt

robots.txt grants or blocks crawler access. llms.txt describes your site for language models. You usually want both, and they do not replace each other.

Updated 2026-09-06

Different jobs

robots.txt is an access-control file. Crawlers look it up before fetching other paths and honor Allow / Disallow rules for their user agent. If PerplexityBot is disallowed, it should not crawl your pages no matter how good your llms.txt is.

llms.txt is a description file. Models that support it can read a short map of your site. It does not allow or block anyone. A beautiful llms.txt behind Disallow: / is a brochure on a locked door.

Who reads each file

Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and most other crawlers read robots.txt. That is a decades-old convention.

llms.txt is newer and still optional. Some tools and agents look for it; many do not. Treat it as extra context, not as a substitute for crawler rules. More on whether you need llms.txt.

A practical setup

Keep robots.txt explicit for search/fetch bots you care about, and keep training tokens as a separate decision. Add llms.txt if you want a canonical explanation of the product, with links to docs, pricing, and this robots policy.

amisearchable.cc ships both on purpose: /robots.txt allows AI crawlers, and /llms.txt describes the product. The checker reports both because people conflate them.

What the report shows

A domain report lists each crawler as allowed, blocked, or not specified, then shows whether llms.txt was found. “Not specified” means robots.txt has no rule for that bot, so most crawlers fall back to the * group or treat the path as allowed.

Check your domain

See which of these bots can reach you. No account needed.

Related guides