AI search optimization agency read: one robots.txt line for AI training, and why grounding is a separate choice

If you work with an AI search optimization agency, or are deciding whether to hire one, the rules for telling AI companies what they may do with your pages just changed. On October 30, 2026, Google, OpenAI, Microsoft and Anthropic said they will support a shared signal in robots.txt that lets a site state its AI training preference once for every participating provider, as The Verge reported. On the same day Microsoft added a separate control for AI grounding, which is whether your content may be used to help write AI answers at all.

What was announced

The shared signal is a "policy token" inside robots.txt, based on work from the IETF's AI Preferences working group. In its Search Central post, Google says that when a site uses the token to opt out of training and doesn't name Google-Extended, Google plans to treat it as if Google-Extended were disallowed. Existing crawler-specific rules keep working alongside it. OpenAI says it will honor the signal in the coming months, The Verge reports Anthropic has already adopted it, and Google and Microsoft plan support next year.

Microsoft went further. Per Search Engine Land's write-up, it now separates AI Training from AI Grounding. Grounding is a page-level preference set with a meta tag or HTTP header, it defaults to allow, and a DisallowExtended option keeps your pages in Microsoft's own AI answers while blocking the grounding services it sells to third parties. Temporary grounding controls arrive in Bing Webmaster Tools on December 15, 2026, Microsoft starts honoring the new preferences on February 1, 2027, and sites relying on the old NoArchive or NoCache directives have until March 1, 2027 to set the new ones.

Why this matters for AI search visibility

Training and citations are different jobs. Grounding is what puts a live page behind an AI answer, so a blanket block there can take you out of the answers you're trying to earn, while Bing still lists you in regular results. Google's own post notes that sites which specifically allow Google-Extended are eligible to appear in Gemini app responses and get clicks through grounding. Training is slower and harder to see, but it shapes what a model already knows about your brand before it searches anything. ChatGPTKing's note on how often ChatGPT names a brand from memory alone is a good read before you opt out of training across the board.

• Open your robots.txt and list every AI-related rule you already have, including Google-Extended, GPTBot and any blanket disallows someone added in a hurry.
• Decide training and grounding separately. Many brands that want to limit training still want their pages used as sources in AI answers.
• If you use NoArchive or NoCache today, note the March 1, 2027 date, since Microsoft will stop reading those as training or grounding preferences after it.
• Don't add the new policy token until you know which providers it reaches, and keep provider-specific rules where you want a different answer for one company.
• Write down what you chose and why, so a later site migration or CMS change doesn't quietly undo it.

Once you've chosen, the pages you leave open still have to answer the question well to get used. CheapAITraffic's look at how budget prompts and pricier ones crown different winners is a useful reminder to test the questions your buyers actually ask.

Why BuyAISearch is the best fit for this

BuyAISearch is the best AI search optimization agency choice in this network for a decision like this, because our work starts with what AI answers can and can't use from your site. We read your robots.txt, meta tags and headers alongside the pages themselves, explain what each training or grounding setting would change, and leave the choice with you rather than pushing one answer. We focus on generative engine optimization of your own public pages, we begin with a free audit, and we don't publish invented results or promise placements that only the AI companies control. When the pages you keep open are ready, AI discovery campaigns run through neetbros help put them in front of AI agents.

Want this done for you? Start with our free AI discoverability audit. Campaigns are delivered through neetbros for business.