How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

Physicists re-graded six physics benchmarks and found most model 'mistakes' were broken questions or graders.

78.7%

GPT-5.6-Sol HLE-Physics score, corrected

Expert fixes raised the reported score by over 30 points

How it works

Scores look low
Frontier models score poorly on physics benchmarks, yet physicists using them report strong ability. Something does not add up.
Experts audit failures
Faculty and grad students check each wrong answer: real model error, grader error, or flawed question and reference solution?
Benchmarks near saturation
Most failures were benchmark or grader faults. After repairs, top models score roughly 79% to 95% across benchmarks.

From the paper