Google Shows a Robot That Follows Instructions in Plain English

The program, RT-2, connects a robot arm to the same kind of AI that powers chatbots, so it can act on requests it was never trained on — like picking up the extinct animal from a table of toys, and choosing the dinosaur.

Google's DeepMind lab showed a robot today that takes instructions the way you would give them to a person. Told to "pick up the extinct animal" from a table holding a toy dinosaur, a lion, and a whale, it picks up the dinosaur. Told to move a can of Coke to the picture of Taylor Swift, it does. Nobody trained it on dinosaurs or pop stars.

The program behind it is called RT-2, and the idea is to connect a robot arm to the same kind of AI that runs a chatbot. A chatbot has read the internet and knows that dinosaurs are extinct. A robot, until now, knew only what it had been shown by hand — thousands of demonstrations of specific tasks with specific objects, and nothing beyond them. RT-2 treats a robot's movements as if they were another language, so the same program that understands the sentence can produce the action. In DeepMind's tests it succeeded on about 62 percent of tasks it had never seen before, roughly double the previous system.

It is a laboratory demonstration: one arm, a tabletop, simple objects. It is slow, and it fails often. But it points at the thing that has kept robots out of homes for decades. Building a machine that can move is a solved problem. Building one that understands what you want it to do has not been — and this is the first clear sign that the chatbot breakthrough might carry over.