Press Release

Frontier AI Systems Solve up to 9/10 Problems in First Proof Benchmark; Open-Weight Entrant Solves None

October 14, 2026 — The results of First Proof’s third test of AI on research mathematics indicate that the gap between open-weight and closed systems in this context is substantially wider than the roughly four-month lag that Epoch AI measured across general AI benchmarks in May 2026.

First Proof tested five AI systems on ten research mathematics problems in fields ranging from geometry and topology to statistics and dynamics which were known to have solutions of at most 12 pages. Thirty expert reviewers met at Carnegie Mellon’s Institute for Computer-Aided Reasoning in Mathematics (ICARM) on October 8 and 9 to grade the anonymized solutions for correctness, exposition, and attribution of prior work. This batch introduced negative scoring to incentivize systems to output no answer rather than incorrect answers.

For the first time, the test included a system built on an open-weight model, Kimi K3, released by Moonshot AI earlier this summer. It also included two harnesses designed by academic teams, plus one harness provided by Anthropic, as well as a direct test of OpenAI’s GPT-6 Astra. All testing took place between October 2 and 5, with 24 hours for each system and no human input.

Key findings

  • Frontier systems improved. The ProofCouncil harness and the underlying OpenAI model each answered two more problems correctly than in the previous batch, reaching 9 of 10 and 7 of 10 respectively. ProofCouncil and OpenAI respectively cost $1,888 and $36 ($3,200 and $117 in Batch 2). Each of the 10 questions in the benchmark was solved correctly by some system, setting aside citation and exposition issues.
  • Open-weight systems are far behind. Every submission by the open-weight system contained major errors, at a total cost of about $2,000. Reviewers compared it to older frontier models and found it generally less capable than the leading frontier model of December 2025 (GPT 5.2 Pro).
  • Cost of improved performance. By leveraging more compute, it is possible to significantly improve on the performance of a single query to GPT-6 Astra — but this comes at significant cost, leading to concerns that access to computational resources may become a barrier for basic research in mathematics.
  • Writing and citation still lag. Frontier models cite the literature substantially better than before, but gaps remain. Only 4 of the 10 GPT-6 Astra solutions were judged to have adequate citations; the ProofCouncil harness, which uses Astra as a base model, was able to improve this to 7 out of 10. Exposition remains generally weak: even the top system ProofCouncil had a third of its responses receive negative average scores.

The complete record, including problems, human solutions, AI solutions, source code, referee reports, costs and system logs, is published at 3 pm ET on October 14, 2026, on the Third Batch page.

What comes next

The next batch will take place in the first half of January 2027. Because the gap between frontier systems and the best open-weight model is now very large, Batch 4 will test the two types of systems separately, in order to identify how open-weight systems fail.

Contact

Full report and supplementary materials: Third Batch results

About First Proof

The First Proof Foundation is a 501(c)(3) non-profit. Batch 3 was supported by DARPA, the Survival and Flourishing Fund, the AI for Math Fund and Google.org, and by unrestricted donations from Anthropic and OpenAI used only for scientific and outreach activities. The Foundation thanks Harvard’s Center of Mathematical Sciences and Applications, which hosted the problem-writing session, and ICARM, which hosted the review meeting.

Terms used in this release

Open-weight model
An AI model whose internals are published, so anyone can download and run it.
Frontier model
The most capable systems from the leading AI companies, available only through those companies.
Harness
Software built around an AI model that breaks a problem into steps, checks work and calls the model or models many times. The same model can perform very differently in different harnesses.
Correctness, exposition, attribution
Whether the solution is right, whether it is clearly written, and whether it credits the existing work it relies on.