For the last couple of years, site owners who did not want their writing used to train AI models have faced an unpleasant trade. The same crawler that collected pages for training also collected them for search, so refusing one refused the other. Turning away the training meant vanishing from the results. Most people, reasonably, decided that was too expensive and let it go.
That trade has now been broken apart, and it is worth understanding how, because a good deal of the mechanism works whether or not you use any particular service.
The only universal mechanism is robots.txt, a plain text file at the root of your site that names crawlers and tells them which parts they may fetch. It is the same file that usually points crawlers at your XML sitemap. It has worked for thirty years and it works because the large, reputable crawlers choose to honour it.
That is also its limitation, and the limitation is worth stating plainly rather than being disappointed by it later. A robots.txt file is a request. It cannot verify who is asking, it cannot tell why they want the page, and it does nothing whatsoever about a crawler that ignores it. Publishing a preference and enforcing a preference are different jobs, and only the first can be done with a file.
The second problem is that a crawler is a single thing while its purposes are several. A company that runs one crawler feeding both a search index and a model training set gives you one name to allow or deny. This is what is meant by a mixed-use crawler, and it is the whole reason the trade existed.
Cloudflare separated the question into three, and got the major crawler operators to agree to answer them separately. From 15 September 2026 a domain can express a different preference for each of:
The new setting, Disallow AI Training, is named after what it actually does: it publishes a Disallow directive in your robots.txt. Mixed-use crawlers that have committed to honouring it keep collecting for search and stop collecting for training. Training-only crawlers, including those run by Amazon, Anthropic, Meta and OpenAI, are blocked outright, since separating those crawlers is what makes blocking them possible without collateral damage.
Three mixed-use crawlers are covered: Applebot, Bingbot and Googlebot. To qualify, an operator has to offer a way to opt out of training, a way to opt out of AI summaries, visibility into which pages were made available for training, and an assurance that opting out of training does not affect ordinary search results. That last commitment is the one that makes the whole thing usable, and all three operators state it.
One caveat matters today. Microsoft is still building support for reading a no-training preference from robots.txt, targeted for early 2027. Until then, selecting the setting does not convey anything to Bing that way, and the route to Bing is the NOARCHIVE meta tag in your pages.
The publishing half of this does not belong to anyone. The operators provide their own opt-outs and honour them regardless of how your site is hosted:
Disallow rule for Google-Extended in robots.txt. Google states this does not affect search inclusion or ranking. There is also a toggle in the webmaster portal to keep your content out of generative search results.Disallow rule for Applebot-Extended. Apple states this does not affect search ranking. The nosnippet directive in your pages covers AI summaries separately, and paywalled content can be labelled to keep it out of generated output.NOARCHIVE meta tag, with granular controls and reporting in Bing Webmaster Tools. Microsoft states it does not affect search ranking.Those four lines are available to every site on the internet. Adding them costs nothing, needs no product and takes ten minutes, and if you have been putting this off because you assumed it would cost you traffic, that assumption is what has changed.
What you do not get without a network in front of your site is enforcement. Identifying a crawler, deciding whether it is who it claims to be, blocking it when it ignores the file, and reporting what it did are things only something sitting in the request path can do. If you are weighing up whether that is worth having, we went through what you get for nothing in the Cloudflare free plan. That is the genuine division here, and it is worth being clear-eyed about which half you are getting.
This is a business decision rather than a technical one, and we would be doing you no favours by pretending the answer is obvious. It is worth working out first what your site actually earns from search, which our introduction to website SEO covers.
The case for opting out is straightforward. If your site is your writing, your photography, your research or your product descriptions, and you would rather it not become training material, you are entitled to that preference and it now costs you nothing in search.
The case against is in the numbers Cloudflare cites alongside the announcement. More than half of people now read the AI summary in search results, and those who do are substantially more likely to stop there rather than click anything. Set against that, visitors arriving through AI search are reported to convert at three to five times the rate of ordinary search traffic, one comparison putting it at 14.2 per cent against 2.8 per cent. Fewer visitors, better ones.
Note that training and summarising are separate questions. Disallowing training does not remove you from the summaries, which are generated from a live fetch. If your concern is that summaries answer the question so nobody visits, training controls are not the lever. The summary controls are, and they are less mature.
For most small businesses, our view is that the summary question matters more commercially than the training question, and the training question matters more on principle. They are worth deciding separately rather than as one sweep.
Notably absent is any way to say no to agents, the bots fetching a page because a person just asked something. That is deliberate: there is no established way to express that preference yet, and work is under way at the IETF to define one. Expect this area to keep moving, and expect anything you configure now to need revisiting within a year.
The general direction is encouraging though. For most of the last two years the only options were all or nothing, with the interesting decisions impossible to express. Being able to say “index me, do not train on me” as a routine setting is a meaningful improvement, and the fact that under a fifth of sites currently block training at all suggests most site owners have simply never had a practical way to act.
robots.txt have always existed, and the answer to those is rate limiting and blocking at the edge, not a politer request.If we host your site and you would like the training opt-out added, open a ticket and we will put it in place and confirm what it does and does not cover. If you are unsure which way to decide, we are happy to talk it through against what your site actually earns from search.