Should You Block AI Training In robots.txt? What It Costs You

If you’ve seen a post this month telling you one line in robots.txt will stop AI companies using your website, you’re not alone. It’s doing the rounds, and it’s only half right.

For most businesses, the answer is no, not yet. The shared opt-out doesn’t start until 2027. On Google it doubles as a block on Google-Extended, and Google says only sites that allow Google-Extended are eligible to appear and get clicks in Gemini Apps. Measure what AI answers already send you, and leave robots.txt alone while the November core update rolls out.

Part of our AI Search Guides: plain-English guides to how Google's AI and ChatGPT pick websites.

Key Facts

• Google said on 30 October 2026 that it plans to support a shared robots.txt control for opting out of AI training, based on a draft from the IETF AIPREF working group (Google Search Central Blog).
• Google says a site that uses the shared opt-out without naming Google-Extended will be treated as if it had disallowed Google-Extended (Google Search Central Blog, 30 October 2026).
• Google’s post states that websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps (Google Search Central Blog, 30 October 2026).
• Google says it aims to support the new control along with other AI providers in 2027 (Google Search Central Blog, 30 October 2026).
• Search Engine Land reported on 30 October 2026 that Bing, OpenAI and other AI companies back the same IETF proposal.
• Google warned on 11 November 2026 that blanket CDN or firewall bot protection can leave its crawlers stuck on an error page, and that Googlebot cannot solve CAPTCHAs (Google Search Central Blog).
• Google’s November 2026 core update began on 10 November 2026 and may take up to two weeks to complete, according to Search Engine Journal.

This article explains what the new shared AI training opt-out is, how training differs from grounding and display, what an opt-out costs on Google compared with Bing, what two of our clients would be putting at risk, and what to check before you change anything.

What Is The New Shared AI Training Opt-Out?

It’s a single robots.txt signal that tells every participating AI company not to train its models on your content, instead of blocking each one by name. Google announced its support on 30 October in “Supporting a shared AI training control for websites”, written by John Mueller.

The control is based on a draft from the IETF’s AIPREF working group. It builds on Google-Extended, which has been Google’s training control since 2023.

Google isn’t acting alone. Search Engine Land reported that Bing, OpenAI and other AI companies back the same proposal. According to heise online, the token is __usage_ai-training and sits in the ordinary User-Agent line, so robots.txt gets no new commands.

Google’s post also says the proposal fits the EU AI Act’s Code of Practice for general-purpose AI models, which treats robots.txt as a main way for publishers to state their preferences. That tells you where this is heading. Robots.txt is becoming the place regulators expect you to say no.

So far, so tidy. The catch is in what “training” actually switches off on each platform.

Isn’t Blocking Training The Same As Blocking AI Answers?

No. There are now three separate switches, and most owners we speak to treat them as one.

• Training. Whether your content teaches the AI model. Set in robots.txt: Google-Extended today, the shared token from 2027.
• Grounding. Whether an AI tool reads your page live to build an answer and links to you as the source. This is the one that can send visits.
• Display. Whether your pages appear in Google’s AI Overviews and AI Mode at all. That’s Google’s Search Console switch, which we covered in our 24 October post on opting out of AI Overviews.

Training is about the past: what the model learned. Grounding is about today: who gets quoted and clicked when a buyer asks a question.

For a business that wants enquiries, grounding is the switch that pays. The trouble is that Google and Bing don’t draw the line between training and grounding in the same place.

What Does A Training Opt-Out Cost You On Google Compared With Bing?

On Google it can cost you eligibility for clicks in Gemini Apps; on Bing, from what’s been reported, it shouldn’t touch your place in results.

What it looks like: Google says that if you use the shared token and don’t name Google-Extended, it will treat your site as if you had disallowed Google-Extended. Company-specific controls keep working alongside the shared token.

Key data: One sentence in Google’s announcement matters more than the rest: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” Read it the other way round. Block Google-Extended, directly or through the shared token, and you give up that eligibility.

Why it matters: Microsoft took a different route. Search Engine Land’s coverage of Bing’s 30 October announcement says Microsoft gives publishers separate opt-outs for AI training and for AI search grounding, while keeping them eligible for traditional Bing results. On Bing you can refuse training and still be quoted. On Google, the “training” switch reaches into an answer engine.

What about Google Search itself? Google frames the shared token entirely in terms of Google-Extended, its training control. Before you assume your rankings and AI Overviews are untouched, read Google’s crawler documentation on Google-Extended, which is where Google sets out what each token does. Do the same with OpenAI’s own crawler documentation before you block any of its bots by name.

What Would A Business Actually Be Switching Off?

For many businesses, a growing stream of AI-answer links and the visits that come with them. Our own case studies show how much that can be.

Since starting with SEOValley in December 2024, an Australian home builder has more than doubled its top-5 Google rankings. AI answers in Google AI Overviews, AI Mode, ChatGPT, Perplexity and Gemini now link to it as a source 2,868 times.

Its top-5 keywords rose from 2,377 to 4,974 (+109%). Google’s AI Overviews link to its pages for 2,108 different searches, and an estimated 2,270 monthly visits come from AI answers. You can read how an Australian home builder became a source AI answers link to 2,868 times.

The lesson is about how its buyers behave. They research for months before they call, so being the source AI answers link to is part of how leads arrive. Gemini is on that list of engines. The case study doesn’t split the links by engine, so I won’t pretend to know what a Google-Extended block would cost in numbers. What I can say is that it would put part of that visibility at risk, for a benefit this business doesn’t need.

Retail tells the same story. For a UK car lighting retailer, AI answers in Google and ChatGPT now link to its pages 1,956 times, and organic search visits rose 33%, from 62,923 to 83,654, comparing two 12-month periods. The details are in the UK car lighting retailer case study, and there are more examples in our case studies.

Then there’s the wider context. Ahrefs found AI Overviews go with a 58% lower average click-through rate for top-ranking pages, in a study of 300,000 keywords comparing December 2023 with December 2025 (published in May 2026). When the classic blue-link click is shrinking, the links inside AI answers are the last ones you want to throw away.

Should You Add The Token Now Or Wait Until 2027?

Wait. There’s nothing to add yet, and when there is, it should be a decision rather than a reflex.

Google’s post says: “We aim to support this new control along with other AI providers in 2027.” According to Digital Applied’s reading of Google’s documentation in early November, Google’s crawler page still lists Google-Extended as the working training control, with only a note that the shared token is planned for early 2027. Check Google’s crawler page for the current wording before acting. The IETF proposal is still a draft, and drafts change.

There’s a timing reason too. Google’s November 2026 core update started on 10 November, according to the Google Search Status Dashboard, and Google says it “may take up to 2 weeks to complete” (as quoted by Search Engine Journal).

If you edit robots.txt mid-rollout and traffic drops, you won’t know whether it was the update or your edit. One change at a time. Never during a core update.

Could Something Already Be Blocking Crawlers Without You Knowing?

Quite possibly, and for most businesses that’s a bigger risk than any training opt-out.

What it looks like: You switch on bot protection at your CDN, firewall or host to stop scrapers. It works. It also catches the crawlers you need.

Key data: In its 11 November post on moving to a CDN without overwhelming your origin or blocking Googlebot, Google warns that blanket bot protection without exceptions for verified crawlers leaves its crawlers “stuck on an error page that’s not useful for Search.” Googlebot can’t solve CAPTCHAs.

Why it matters: In our experience, an accidental block is a far more common problem than a deliberate AI opt-out. Nobody decided to block anything. A security setting did it quietly, and the first sign is often a traffic drop months later.

Before you debate the principle of AI training, confirm that Googlebot, Bingbot and the AI search crawlers you actually want can reach your pages. If you’re not sure, a technical SEO review of robots.txt, CDN and firewall rules is the sensible place to start.

How Can You See What AI Answers Already Cite You Before Deciding?

Look at the reports the search engines now give you, and do it before you touch robots.txt.

• Bing Webmaster Tools, AI Performance. On 13 October Bing added Topics Portfolio (beta), which groups grounding queries into topics and shows queries where other sites are cited but yours isn’t. That’s a direct view of what grounding is worth to you, and what you’d be giving up.
• Google Search Console. Its generative AI performance reports cover the Google side. Look at the trend, not a single week.

If the numbers are tiny, an opt-out costs you little. If they’re growing, you have your answer. We’ve written a fuller guide on measuring AI visibility when there’s no rank to track.

When Is Opting Out Of AI Training The Right Call?

When your content is the product. Publishers selling original reporting, paid research or proprietary data have a real case: if a model learns from your work, it can answer the question your subscription was meant to answer.

Even then, think in parts. Bing reportedly lets you split training from grounding. Google says the shared token only counts as a Google-Extended block if you don’t name Google-Extended, and that company-specific controls keep working alongside it. So a publisher could opt out of training broadly while making a separate, deliberate choice about Google. How that plays out depends on how much Gemini matters to your audience.

For a builder, a retailer or a B2B supplier, my view is simple. You want to be quoted. Blocking training to protect content whose whole job is to win customers is like taking your shop sign down so nobody copies the font.

My advice for the next few weeks: leave robots.txt alone until the November core update has finished, check that your CDN and firewall aren’t blocking crawlers you need, and pull your Bing AI Performance and Search Console data so you know what grounding is actually worth to you. Then look at the shared token again once Google and the other providers start supporting it in 2027.

If you’d like a second opinion on where your site stands in AI answers before making that call, that’s the work our AI SEO service focuses on.

Frequently Asked Questions