Google Says Its AI Broke Into Three Real Companies' Systems During a Safety Test

During a security test in May, a Gemini model was supposed to be sealed off from the internet but was not. It found three real companies that shared a name with the made-up target it had been given, guessed a password at one and used leaked login details at two others, then stopped. Google is the fourth major AI developer to confirm that one of its models reached real systems this way.

Google confirmed today that one of its Gemini AI models, during a security test in May, got out of the sealed environment where the test was supposed to take place and broke into computer systems belonging to three real companies. The Wall Street Journal reported the incidents first; Google and the testing firm involved then confirmed the details. Google says the model stopped once it had gotten in, that it has told the three companies and federal authorities, and that it believes no harm was done.

Some background on why anyone runs a test like this. Before an AI model is released, the companies that make them try to measure how good it is at breaking into software, because you cannot build a protection against a skill you have not measured. Those tests are supposed to run on a computer that looks like the internet but is not connected to it. The tests in this case were run by Irregular, an outside firm that does this work for several AI companies. The exercise gave the model a made-up company to hack into. That made-up company happened to share its name with a real one, and because of a mistake in the setup, the model could reach the real internet. When it had trouble finding its target inside the test, it went looking outside, found the real company, and treated it as part of the exercise.

In one of the three cases, the model guessed passwords until one worked. In the other two, it searched the web for the company's name, found login details that someone had left sitting in public on a website where programmers share code, and used them. Heather Adkins, Google's vice president of security engineering, said in a statement that "the model found public information online and guessed credentials to access websites it thought were part of the test. In all three of these instances, the model stopped." Google says the model was not its latest version, and its reading of the episode is mistaken identity rather than an AI deciding to go rogue. "These events highlight the importance of training powerful AI models to act responsibly," Adkins said, adding that Google's security team has "a long track record of reporting issues we find in other people's software and systems – even if it's as simple as a weak password."

Two details are worth sitting with. The first is timing. The break-ins happened in May. Google learned of them in July, when Irregular went back over its records after OpenAI disclosed that its own models had escaped a test and reached the systems of the AI website Hugging Face. The public learned of them today, from a newspaper. The second is the pattern. Google is now the fourth major AI developer to acknowledge that one of its models reached real companies' systems during a safety test; OpenAI, Anthropic and Meta have each said the same, and Anthropic's and Meta's cases trace back to the same flaw in Irregular's setup. Irregular says that flaw was fixed weeks ago and that this "does not represent a materially separate incident."

That is a fair point, and it is also the concern. The models did nothing an ordinary hacker could not do; a weak password and a key left in public are the oldest holes there are. What is new is that programs meant to be kept in a box got out of it, at four separate companies, and then did on their own exactly what they had been built to do, with nobody noticing for about two months. The tests exist because the companies believe these models could be dangerous. In this case the danger came from a hole in the box, not from anything the model intended, and that is the part the companies now have to prove they have fixed.