Your robots.txt is probably blocking the crawlers you want

Nobody sets out to hide from the engines. They inherit a template, or ship a staging file, and quietly stop being readable by the thing writing the answers.

The cheapest way to be invisible

Everything else in this library is work. This one is a file with a handful of lines in it, and getting it wrong undoes all the rest — a site that cannot be fetched cannot be quoted, however good the pages are.

Nobody chooses this. They inherit a template from a theme, or ship a staging config to production, or paste in a block from an article about blocking AI scrapers without noticing which agents it names. Then they spend a year wondering why their site never turns up in an answer.

Ten minutes, once. Start by opening yourdomain.com/robots.txt and actually reading it.

The agents worth knowing about

These are the ones that matter for answer engines. They do three different jobs, and conflating those three is the source of most of the damage.

| Agent | What it is for | | --- | --- | | GPTBot | Crawls pages for model training | | OAI-SearchBot | Builds the search index behind ChatGPT’s answers | | ChatGPT-User | Fetches a page live, because a user’s question needed it | | ClaudeBot | Crawls pages for model training | | Claude-User | Fetches a page live for a user’s request | | PerplexityBot | Builds the index Perplexity answers from | | Perplexity-User | Fetches a page live for a user’s request | | Google-Extended | Controls use of your pages by Google’s AI products | | Applebot-Extended | Controls use of your pages for Apple’s model training | | CCBot | Common Crawl, an open corpus many models are built on |

The list moves. Providers add agents, rename them and split one into two, so treat this as the shape of the problem and check the current documentation before you write a rule about a specific name.

Three jobs, one file

Training crawlers read your pages to build a model. Blocking them keeps your words out of a future model’s weights. It has no direct effect on whether you are named in an answer today, because today’s answers about local businesses come mostly from live search rather than from recall.

Search crawlers build the index the assistant searches when somebody asks a question. Block these and you are removing yourself from the retrieval step. This is the one that costs you customers.

User-triggered fetchers go and get a specific page because somebody asked something that needed it. Blocking these means the engine cannot read the page it was about to cite, in the moment it wanted to cite it.

Two of those three are how you get named. The public argument about AI and copyright is almost entirely about the first one, which is why so many blanket rules copied from that argument cause damage their authors never intended.

The decision you are actually making

Opting out of model training is a legitimate position. Publishers, photographers and people with a large body of original writing have real reasons for it, and there is no reason to be sniffy about it.

Opting out of search is different. It is not a stance about intellectual property; it is a decision to be absent from the place your customers are asking. For a local business with a handful of pages describing services in a town, the trade is lopsided: the training value of your site is negligible to anybody, and the cost of being unreadable at the moment of the question is a lost job.

So the common configuration for a local business is: allow the search and fetch agents, decide about the training ones on your own terms. What you should not do is block all ten because a plugin offered a checkbox labelled block AI bots.

Reading the file you already have

Two rules of precedence explain most confusing files.

A group naming an agent explicitly wins outright over the wildcard group. A file that disallows everything under User-agent: * but has a group for OAI-SearchBot allowing it is correctly read as open to that agent — the specific group applies and the wildcard is ignored entirely for it.

And an empty disallow means allow. Disallow: with nothing after it is the conventional way of saying nothing is blocked, while Disallow: / blocks the whole site. Those two lines look nearly identical in a diff and mean opposite things.

Here is a file that does the sensible thing for a local business.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

That is genuinely it. A permissive site does not need per-agent rules at all, and every rule you do not write is a rule that cannot be wrong. Add specific groups only when you actually want a specific agent treated differently:

# Allowed to answer questions, not to train
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The four ways this goes wrong

The staging file shipped. A Disallow: / written to keep a preview site out of search, deployed to production with everything else. This is the most common and the most expensive, and it usually survives for months because nothing on the site looks broken.

The template you inherited. Themes and site builders ship default files, and some of them block generously. Read what yours actually says rather than assuming it is empty.

The checkbox. A plugin or a platform setting offering to block AI bots, implemented as a blanket rule across every agent in this article. The intent was about training. The effect includes search.

The layer robots.txt cannot see. Plenty of sites are fronted by a firewall or bot-management service that blocks unfamiliar agents before the request ever reaches your file. A perfect robots.txt behind a rule that returns a challenge page to everything non-human is still an unreadable site. If your file looks right and nothing is reading you, check that layer next.

Declare your sitemap while you are here

One line, and it is the cheapest point in the whole readiness half of a visibility score. It tells a crawler where the list of your pages is, rather than leaving it to discover them by following links from the homepage.

Two things to verify. That the sitemap is declared in robots.txt, and that the URL you declared actually returns a sitemap. A great many sites answer /sitemap.xml with a cheerful 200 and their homepage, because the framework serves the homepage for anything it does not recognise — which is indistinguishable from a working sitemap unless somebody looks at the body. Open it and confirm you see a list of URLs.

The same trap catches llms.txt, for the same reason.

Allowing one agent while blocking another

This works, and it is worth understanding why the two halves are separate in the first place.

Each provider publishes its own agents precisely so that a publisher can make different decisions about different uses. The extended agents are the clearest example: Google-Extended governs how your pages may be used by Google’s AI products and has nothing to do with whether ordinary Google Search indexes you, and Applebot-Extended is the training opt-out that sits alongside the ordinary Applebot that powers search. Blocking the extended one does not remove you from the search product, and confusing the two costs people search visibility they never meant to give up.

The corollary is that these rules are honoured by the vendor who publishes them, on their own terms, and each vendor defines its own agents. There is no single switch for all of this and anybody selling you one is simplifying something that is not simple.

Per host, per scheme, per subdomain

A robots.txt applies to exactly one host, and this catches people out.

Your apex domain and your www subdomain are different hosts as far as the rule is concerned, even when one redirects to the other. If a booking system, a shop or a landing-page builder lives on its own subdomain, that subdomain has its own file — and hosted platforms very often ship a restrictive default there that you never chose and cannot see from your main site.

Check each host you actually publish content on. The one carrying your prices or your service descriptions is the one that matters most, and it is frequently the one somebody else is hosting.

Confirming it is really the file

If everything reads correctly and you are still invisible, stop editing the file and go and look at what is arriving.

Your server logs, or whatever request analytics your host provides, will show the user agents that fetched pages. Search them for the names in the table above over the last month. Seeing several of them arriving regularly tells you the file is fine and your problem is elsewhere — usually that there is nothing on the pages worth quoting. Seeing none of them at all, on a site that has been live for a while, points at something upstream: a firewall rule, a bot-management setting, a country block, or a hosting plan that challenges unfamiliar clients by default.

That distinction saves a great deal of time, because the two situations look identical from the outside and have nothing in common.

One more thing worth doing while the file is open: fetch it the way a crawler would rather than in a browser tab. A browser will follow a redirect from the apex to the www host without telling you, so the file you are reading may not be the file an agent gets. Request it directly, note the status code, and confirm the body is the text you expect rather than a styled error page pretending to be one.

What this file cannot do

It is a request, not a fence. Well-behaved crawlers honour it; anything determined to scrape you ignores it entirely, and always has. If you need something not to be read, the answer is authentication, not a line in a text file.

It also cannot remove you from anything retroactively. Blocking a training crawler today does nothing about a model trained last year, and blocking a search crawler does not delete the index entry that already exists — though that will decay.

And it is not a ranking lever. Allowing every agent does not earn you a place in an answer; it only stops you being disqualified from one. Being readable is the floor, and the pillar breakdown is deliberate about how few points it is worth. What it is worth is everything you would have lost.

Check it in ten minutes

Open your robots.txt and read every line rather than skimming for the word bot. Confirm no group disallows everything by accident, that any group naming a specific agent says what you meant, and that a sitemap is declared and returns real content.

Then go and make sure there is something worth fetching when they arrive: the facts, marked up, and pages that answer the question.