OpenAI Posts Hundreds of Math Papers Written by an AI It Hasn't Released, and Mathematicians Say the Checking Has Barely Begun
OpenAI put 722 mathematical write-ups online today, all produced by an AI program it is keeping to itself, and says many of them settle questions mathematicians had left open. Some have been checked by computer and some haven't, and the mathematicians advising the company say releasing them is the start of the work, not the end.
OpenAI today posted 722 mathematical papers on GitHub, a public website where people share computer code and documents, free for anyone to read and reuse. The company says nearly all of them were written by a single AI program it uses internally and has not released to the public. It grouped them into 372 families of related results across many branches of mathematics.
Here is how the company says it got them. It gave the program roughly 4,000 problems, many of them questions mathematicians had not been able to answer, and kept only the answers it judged significant. On average, each result took about three hours of the program's thinking time. For 10 of the results, OpenAI also released shortened accounts of how the program reasoned its way to the answer.
In mathematics, an answer only counts once someone proves it: a step-by-step argument showing it must be true. Checking a long proof can take experts months. Some of these proofs have been run through a program called Lean, which checks every logical step by computer, and many have not. OpenAI says so itself: "This collection includes results at different stages of verification," and "some of the unformalized results could have issues." Even a proof that passes Lean is only as good as the way the original question was typed into it.
The release comes a month after OpenAI said its AI had solved one of mathematics' most famous unsolved problems, a claim mathematicians are still checking. Since then, nine mathematicians formed an independent advisory group, hosted at the Institute for Advanced Study in Princeton and unpaid, to advise AI companies on how to share results like these. On September 29 the group recommended that companies publish the exact instructions they give their programs along with a record of the programs' reasoning. Today's release includes the reasoning summaries for 10 results, not the instructions. "We want to create standards and practices so that results released from AI labs can be understood," said Melanie Wood, a Harvard mathematician in the group. Tonight the group said that putting the work out in public is "the beginning, not the completion, of the process of human understanding."
The two sides see the same papers differently. OpenAI says it has "drawn on their advice and public recommendations to inform how we release these results," and its research lead, Dan Roberts, says the proofs are a by-product of testing its programs to build better tools. Some mathematicians object to the sheer volume arriving at once, without the slow process in which other experts review a paper before it's published. "I don't think they've done their sort of due diligence at all," said Tristan Buckmaster, a mathematician at New York University. Nobody outside the company can run the program to see whether it would produce the same results.
Sources
- OpenAI — openai/math repository of AI-generated manuscripts (GitHub)github.com · primary
- The New York Times (via The Boston Globe) — OpenAI releases a new batch of AI math resultsbostonglobe.com
- Decrypt — OpenAI says a secret AI model cracked hundreds of open math problems; mathematicians want receiptsdecrypt.co