Artificial intelligence models tasked with forecasting scientific outcomes frequently agree with one another far more closely than they align with actual laboratory measurements, according to a newly published evaluation. While pooling the rankings of multiple AI systems can improve average predictive performance, researchers found that shared biases and systemic errors can easily survive the consensus-building process, raising critical questions about the reliability of automated scientific forecasting.
The study, titled the Science Prediction Universe and led by researchers at UniPat AI, was submitted to the arXiv preprint repository on Nov. 11. The comprehensive evaluation tested 13 distinct agent configurations—encompassing 12 foundational models, various reasoning settings, and specialized agent software—across 100 public tasks and 16 private tasks. The evaluation challenged these systems to rank specified sets of experiments within complex scientific domains, specifically targeting chemistry, materials science, biology, and medicine.
Rather than asking the models to forecast exact numerical values, the benchmark evaluated how well the AI configurations could order experimental outcomes. For instance, the agents were tasked with ranking reaction conditions by chemical yield or sorting materials by their electrical conductivity. In this ranking framework, a perfect ordering without measurement ties scores 1, while entirely random rankings average 0.
According to the Science Prediction Universe researchers, the average agreement between scored configurations on the public tasks reached a Spearman rank correlation of 0.52. By contrast, the average correlation between the AI predictions and actual measured outcomes stood at a much lower 0.33. In fact, pairwise agreement between different AI agents exceeded their correlation with empirical laboratory measurements on 81 out of the 100 public tasks. These aggregate figures were calculated using available scored predictions, with data coverage varying across different model configurations.
Consensus Improved Predictions Despite Shared Flaws
Despite the tendency of AI models to form insular agreement bubbles that stray from empirical reality, the study uncovered a distinct mathematical benefit to pooling predictions. By averaging each experiment’s relative position across multiple configurations, the researchers generated a consensus ranking that outperformed the average individual submission.
Specifically, the consensus ranking scored 0.08 points higher than the average submission, yielding measurable improvements on 73 of the 100 public tasks. The researchers reported a 95% confidence interval for this performance gain ranging from 0.05 to 0.11.

However, the study also revealed the limitations of relying purely on collective agreement. On a specific oncology-related task, for example, eleven different configurations achieved a remarkably high internal agreement score of 0.9, yet averaged a mere 0.27 when compared against actual laboratory measurements. Furthermore, the benchmark’s single best-performing individual system managed to outperform the collective consensus of its peers on matched tasks, demonstrating that group agreement is not a foolproof guarantee of superior accuracy.
More broadly, the data indicated that tasks exhibiting higher levels of inter-model agreement also tended to produce better predictions overall. The study’s authors frame consensus as an informative yet inherently fallible signal. They explain that averaging predictions across diverse systems can successfully iron out errors that vary from one model to another, but it ultimately preserves common mistakes that multiple systems share.
The researchers traced these persistent shared errors back to simplistic underlying assumptions regarding how experimental outcomes should behave—such as linear expectations of how variables should rise or fall. If several distinct AI models rely on the same flawed relationship or heuristic, aggregating more of their opinions will only reinforce that shared mistake rather than correct it.
Technical Hurdles and Evaluation Scope
The execution of the benchmark also underscored practical challenges in deploying AI agents for scientific tasks. The 13 configurations were evaluated across differing sets of tasks due to operational constraints and safety protocols. For instance, advanced models such as Claude Opus 5.5 and Claude Fable 5.1 produced scores on only 44 and 30 public tasks, respectively, largely as a result of safety-related refusal triggers. Across the entire benchmark, all 13 configurations shared a common dataset of just 28 scored tasks, comprising 27 public tasks and a single private task.
Furthermore, the public tasks examined retrospective data from already published measurements and were carefully selected following initial pilot evaluations. To test generalization further, the researchers incorporated a private set containing 16 tasks spread across four unpublished experimental series, though this smaller sample size inherently limits the broad applicability of those specific findings.
Even so, the private dataset demonstrated meaningful predictive capability. According to results released by UniPat on Nov. 12, 12 of the 13 configurations successfully ranked the unpublished experiments significantly better than chance overall. Out of 208 official private runs attempted during this phase, 159 were successfully scored.
The study did not incorporate a human prediction baseline, leaving open questions about how expert human scientists would perform on the same ranking tasks. The findings apply strictly to the specific experimental outcomes tested within the benchmark framework, underscoring that empirical laboratory measurements remain an indispensable step for verifying scientific hypotheses, regardless of how uniformly artificial intelligence systems agree in their projections.
