Anthropic Says Its AI Wrote Computer-Checked Proofs That 123 Widely Used Pieces of Free Software Can't Crash, Fixing Hundreds of Bugs Along the Way
Anthropic, the company that makes the Claude chatbot, says Claude wrote mathematical proofs, each checked by a separate computer program, showing that 123 pieces of free software used inside countless other programs will not crash on any input below a set size. Along the way it found and fixed hundreds of bugs that ordinary testing had missed. The guarantee covers one particular setup, and the company says plainly that a proof is not the same as bug-free.
Anthropic, the company behind the Claude chatbot, published research today in which Claude wrote mathematical proofs about real, widely used software. Working with a small company called Theorem, it covered 123 open-source libraries: free, shared pieces of code that thousands of other programs are built on. Among them are libraries for reading a common data format, for shrinking files, for printing, and for connecting to a computer remotely.
The usual way to look for bugs is testing: throw huge numbers of random inputs at a program and see whether it breaks. That finds many problems, but it can never try every possible input. A proof is different. It is a step-by-step argument that something holds for every input, and here each proof was checked by a separate program that accepts it only if every step is valid. Anthropic says Claude proved that the parts of these libraries it examined, 239 in all, "will always exit normally" without the kinds of errors that cause crashes or security holes, for any input smaller than 64 megabytes, roughly the size of a short video.
The proofs are enormous, about 180 million lines in total, and took a few weeks to produce. In the process Claude found what Anthropic calls "hundreds of bugs" and patched them, so the proofs describe the repaired software. Some bugs only appeared with unusually large or oddly shaped inputs that ordinary testing would rarely produce: one in a file-shrinking program needed an input of about 20 megabytes, another in an image reader needed 64 megabytes and more than two gigabytes of memory. Anthropic says it is publishing some of the proofs and holding back others that might point attackers to bugs not yet fixed in public versions.
The company lists the limits itself. "Formally verified" does not mean bug-free: the proofs rest on computer descriptions of how the chip and operating system behave, and a mistake there would undermine them. They cover one particular kind of chip and one way of building the software, not the versions running on most people's computers and phones. They only rule out crashes, not wrong answers. Anthropic also notes that Claude could, in principle, "cheat" by changing a program until it was easy to prove; it says it penalized and checked changes to guard against that. The work is also costly in computing power. Anthropic has invited outside experts to check the published proofs; as of today, the claims are the company's own.