OpenAI's New Writing Program Is a Hundred Times Bigger Than the Last One

GPT-3 was trained on far more text than any program before it, and it can do things it was never specifically taught — answer questions, translate, write code — from a few examples typed into a prompt. For now, only researchers can try it.

OpenAI published a paper today describing GPT-3, the successor to the writing program it partly withheld last year. The headline number is its size: 175 billion "parameters," the internal settings a program adjusts as it learns, which is more than a hundred times larger than its predecessor and ten times larger than anything else published. It was trained on essentially the text of the internet, plus digitized books.

The surprise in the paper is not that a bigger program writes better, though it does. It's that size alone gave it abilities nobody trained into it. Show GPT-3 two or three examples of a task — a few English words with their French translations, say, or a few questions with answers — and it does the task on the next one. It can add numbers, unscramble words, answer trivia, and write a passable news article, all from the same program with no special training for any of them. The researchers call this "few-shot learning." Earlier programs needed thousands of examples for each new skill.

It's also, the paper is careful to say, not reliable. It loses the thread in long passages, contradicts itself, and confidently states things that aren't true. On some tests it does no better than chance.

There is no product to try. OpenAI says it will make the program available to outside developers through a paid service later this year, rather than releasing it outright. What people build with it is the open question — the paper reads like a description of a raw material rather than a finished thing.