Is AI ready to rewrite our understanding of physics?
Yale researchers say bad tests, not weak models, explain low scores, and AI is already running lab experiments
Physicists went through 250 answers that AI models got "wrong" on physics benchmarks. Only 12 were genuine physics errors by the model.
That's the headline result of a 2026 preprint from Yale University researchers and colleagues, who re-graded six widely used physics tests. Scores for cutting-edge models had been as low as 30%. After the flawed questions were fixed, they climbed as high as 94%.
As John Sous of Yale points out, the issue was found in the tests and not in the AI. Looking into the questions, errors, ambiguous definitions, incorrect references, and graders grading correct responses incorrectly were found.
One top-performing model went from around 32% on the CritPt benchmark to 94.4% pass4 in the remaining 54 challenges. The revised scores are based on retained subsets and therefore cannot be compared directly with the original. This audit was limited to text-based problems with answers that could be checked.
According to John Sous, AI is going to do extremely well on any problem that is well-defined, no matter how difficult it is.
The messy frontier is another matter. Sous says interesting physics starts when we meet something confusing and build a model to explain it. That's a "very artistic" process, he says. Whether AI can run research end to end is, in his words, "not clear yet".
Mathematics moved faster partly because of formalisation, which turns ideas into code that can be checked. Physics has only begun to use it.
Logan Wright of Yale University teaches a class about AI in physics research, and his syllabus is continuously evolving. His idea is to create experiments in which each input and output is machine-controlled and then transfer control of the machines to AI. As he puts it, the model "interacts with the world through language and code only."
In one of the experiments, the team studies the impact of various shapes of laser pulses on a rubidium atom. Anthropic's Claude controls the pulses and collects data.
An experiment can be completed in one night and produce a week's worth of human work, and suggestions can be submitted via a phone application.
Such experiments will most likely remain within the bounds of constrained domains. Wright believes that soon enough, there will be an experiment designed by AI and incomprehensible to humans. It remains to be discussed whether results incomprehensible to humans are scientific.
