Google Researchers Describe a New Way for Computers to Handle Language

A team at Google has posted a paper describing a design they call the Transformer, which lets a program weigh which words in a sentence matter to each other. It translates English to German better than existing systems, and trains faster.

Eight researchers at Google posted a paper today with a title that sounds like a slogan: "Attention Is All You Need." It describes a new design for the programs that translate between languages, and it will be presented at a machine-learning conference later this year. Outside the small world of people who build these systems, almost nobody will notice.

The idea is a way for a program to decide, for every word in a sentence, which other words it should pay attention to. When you read "the animal didn't cross the street because it was tired," you know "it" means the animal. That's obvious to a person and hard for a machine. Earlier systems read sentences one word at a time, in order, and tended to lose track. The new design looks at all the words at once and works out the connections. The researchers call this "attention," and their design uses nothing else — hence the title.

On the standard tests for translating English into German and French, it beats every previous system, and it does so with far less computing time, because looking at everything at once is easier to spread across many chips than reading in order.

The paper says nothing about chatbots or writing or anything beyond translation. For now, it's a better way to do one job. Whether the same trick is useful for other kinds of language work is a question the paper leaves open.