Claude mathematically proved 123 open-source libraries can't crash, and found bugs trillions of tests missed

Testing can show that software has bugs. It can never show that it has none. Formal verification can: you describe what the code must always do as a mathematical rule, then prove the code obeys it for every possible input.

The problem has always been cost. Anthropic's researchers point to the seL4 operating system kernel, which took two person-years to build and 20 person-years to formally verify. That is why proofs have mostly been reserved for things like aircraft software.

A new study from Anthropic's Frontier Red Team, done with the startup Theorem, argues that AI has changed the math.

What Claude proved

The team asked Claude to prove security properties for 123 open-source libraries. Specifically, Claude showed that their programs always exit normally, with no memory-safety errors and no failed checks, for every input under 64 MB.

Two details make this unusual:

• The proofs work on machine code. Claude proved the properties on the compiled RISC-V assembly, not just the source code, so compiler quirks can't sneak a bug past the proof.
• The specs came from existing tests. Writing down what "correct" means is usually the hardest part. The team turned each project's fuzz tests into a rule that says, in effect, "no input can ever make this crash."

To the researchers' knowledge, this is the first large-scale use of AI to formally verify production software at the assembly level. They are publishing a subset of the proofs so anyone can check and reuse them.

The bugs it found anyway

Proving code correct means hunting down every case where it isn't. Along the way Claude:

• found bugs that had escaped detection despite trillions of test runs,
• proved that the Linux kernel's zlib compression code matches its official specification, turning up a few minor, unexploitable corner-case bugs,
• found a logic bug latent for more than a decade in a widely used application,
• found as many security bugs in one week of work on a popular PDF reader as are typically reported in a year.

In another test, Claude wrote a Python interpreter in Rust and proved it follows Python's official behavior. Across 1.7 million random programs it matched standard Python in 99.95% of cases.

What's still hard

The biggest limit is cost: a single proof can take hundreds of millions of tokens. Two trends help. Anthropic says its newer Opus 5.5 model needed about 40% fewer tokens than Opus 5 on a set of 36 test tasks, and agents that could copy each other's finished proofs used 17% fewer tokens than agents working alone.

The researchers also add a warning worth repeating: "formally verified" does not mean "bug-free." A proof only shows the code matches the rules you wrote. If the rules, or the model of the CPU and operating system, don't match reality, bugs can remain.

Why it matters

AI is making it cheaper to find and exploit bugs, a point Anthropic also made when it launched its Cyber Mission. Cheap proofs are a way to flip that: prove the shared libraries everyone depends on once, and every project that uses them benefits. Theorem has offered to run the same pipeline for other open-source projects.

Sources

• LLMs are making formal software verification feasible at a much larger scale (Anthropic)