OpenAI Shows a Program That Draws Pictures From a Written Description

Type "an armchair in the shape of an avocado" and DALL·E produces a dozen images of exactly that — objects that don't exist, drawn in a style you choose. It is a demonstration, not a product; the lab has not released it.

OpenAI, the San Francisco lab known for its writing program, showed off a new one today that draws. It's called DALL·E — a mash-up of the painter Salvador Dalí and the Pixar robot WALL-E — and it makes images from a sentence. The examples on the lab's website include "an armchair in the shape of an avocado," "a store front with the word 'openai' written on it," and "a baby daikon radish in a tutu walking a dog." Each produces a grid of pictures, and most of them are recognizably the thing described.

The program is a version of GPT-3, the text program from last year, taught to treat images as a kind of language: it learned from millions of pictures paired with their captions, and it produces a picture the way the other program produces a sentence, one small piece at a time. Ask for the same thing in a different style — "as a sketch," "in the style of a 1990s video game" — and it obliges.

The pictures are small, and the failures are easy to find: ask for a specific number of objects and it may lose count; ask for text and it produces letter-shaped gibberish. OpenAI is not releasing the program, and says it is studying how such a thing might be misused before deciding what to do with it.

It arrives alongside a second program, called CLIP, that does the reverse — looks at a picture and describes it — and which the lab says it is releasing. Together they suggest a direction: programs that handle words and images as the same kind of thing.