Anthropic's Claude Can Now Use a Computer: It Moves the Mouse, Clicks Buttons, and Types

Given a task, the program looks at screenshots of the screen and operates the computer the way a person would — opening apps, filling forms, browsing websites. Anthropic calls it experimental and 'at times cumbersome and error-prone.'

Anthropic released a feature today that it calls computer use. Instead of answering a question, Claude can be given a task — book this, fill in that, find the price of this on that site — and do it by looking at the screen, moving the cursor, clicking, and typing, the way a person sitting at the keyboard would. It takes screenshots to see what happened and decides what to do next. In the company's demonstration, it filled out a form by finding the information in a spreadsheet and a web page, switching between them as needed.

This is the first time one of the major labs has released a program that acts on a computer rather than producing text for a person to act on. Anthropic is careful about it. The feature is available only to developers, not in the consumer chatbot, and the company says the program is "still experimental — at times cumbersome and error-prone." On a standard test of computer tasks it succeeds about 15 percent of the time, compared with about 70 percent for people. It is slow. It can be tricked by things it reads on a web page.

But the direction is unmistakable, and the other labs are known to be working on the same thing. A program that can use a computer can, in principle, do any job that is done on a computer. That has been the promise — and, depending on your job, the threat — since ChatGPT appeared. Today it stopped being hypothetical and became a product, with a 15 percent success rate and a warning label.