Google Can Reportedly Use Content to Train Search AI Even If Publishers Opt Out

Google Search products can reportedly use content from publishers even if they have opted out of artificial intelligence (AI) training. As per the report, a Google DeepMind executive revealed the information during a testimony in the company's ongoing antitrust case against the US Justice Department. The executive reportedly highlighted that such content is not used in the AI models developed by DeepMind. The Mountain View-based tech giant reportedly explained that content for search is managed by a separate mechanism that uses the robots.txt web standard.

Update: After publishing the story, Google's global PR team reached out to Gadgets 360 with additional details. As per the statement, the company's publisher opt-out rule only applied to its Google-Extend product, and has never applied to Google Search. Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

Google Follows Different for AI Models, Search Products

According to a Bloomberg report, Eli Collins, the Vice President of Product at Google DeepMind, confirmed that the rules for adhering to publishers' decision to opt out from AI training are different for AI models from DeepMind and the company's Search products.

Google-Extended, which does not and has never applied to Google Search.

Attorney representing the Department of Justice in the antitrust case, Diana Aguilar, reportedly produced a document highlighting that 80 billion out of 160 billion tokens used to train Google's AI models came from content that publishers had opted out of AI training. Collins reportedly responded that DeepMind's models do not use the content once a publisher has opted out of AI training.

However, when Aguilar reportedly questioned if the Gemini AI model could use the same content if it was put inside the Search product, Collins confirmed that as “correct,” as long as the use case was within Search. Notably, this would include Gemini models powering Google's AI Overviews and recently launched AI Mode.

This means traditional opt-out methods aren't enough to keep Google from using content from publishers. The tech giant had updated its privacy policy in June 2023 to reflect that it will use all publicly available Internet data to train its language models. Here, publicly available Internet data refers to any website that does not have a paywall or mandatory sign-up pages, restricting its access to the public.

A Google spokesperson later told Bloomberg that the rules for Search-based AI tools are different, as publishers can “only decline having their data used in Search AI if they opt out of being indexed for search.” Publishers can do this by disabling the robots.txt web standard that allows Google's crawler bots to access the content to index it in search results.

However, this would also ensure that these web pages do not show up when a user uses Google's search engine to search for a topic. This effectively leaves publishers with no option but to accept the company training its AI models on said data.

The ongoing antitrust case is attempting to prove that Google has a monopoly in the search and AI space. Amit Mehta, a US District Judge presiding over the case, is being urged by the Department of Justice to force the tech giant to sell Google Chrome and to share the data that it uses to generate search results. However, no such measure has been suggested for the company's AI products.