339 AI experts gave four AI benchmark results a 24.6% chance, superforecasters 9.7%
A panel of 339 AI experts gave an average probability of 24.6% to four AI benchmark results that then arrived, and a group of superforecasters gave the same four outcomes 9.7%. The figures come from the Forecasting Research Institute's accuracy review, published on September 23.
FRI's chart of the METR 80% time horizon question, with medians of 3.4 and 3.5 hours for the end of 2026. Image: Forecasting Research Institute
The four are MATH, MMLU, QuALITY and an International Mathematical Olympiad gold medal, all forecast in mid-2022 during FRI's Existential Risk Persuasion Tournament. AI reached IMO gold-level performance in July 2025, five years earlier than the median expert prediction of 2030 and 10 years earlier than the median superforecaster's 2035.
FRI's panel is heavy on professors and lab staff
The institute recruited the sort of people a policymaker would call. Its Longitudinal Expert AI Panel opened with 339 experts: 76 computer scientists, 76 AI industry employees, 68 economists and 119 AI policy researchers.
Of the 76 computer scientists, 41 are professors, 30 of those at top-20 institutions, and 10 are among the 200 most-cited authors in AI. Twenty of the 76 industry panellists work at OpenAI, Anthropic, Google DeepMind, Meta or Nvidia, and 12 of the panel are TIME 100 AI honorees.
The misses did not stop in 2022
On FrontierMath Tiers 1 to 3, the median expert predicted the leading model would reach 31% by the end of 2025 and the median superforecaster 30%; the question resolved at 40.7%. On LiveCodeBench Pro (Hard), medians of 14% and 12% for the end of 2026 were overtaken in May 2026 at 53.8%.
One question came much closer, and then aged in weeks. Asked in April and May 2026 how long a task the best model would handle at 80% reliability by the end of the year, experts answered 3.4 hours and superforecasters 3.5. On May 8, before the survey period had closed, METR listed Anthropic's Mythos Preview at 3.1 hours.
The revenue forecasts were out by a factor of five
FRI also asked for the highest annual recurring revenue at any AI-focused company by the end of 2026. AI experts gave a median of $20 billion, economists $16 billion and superforecasters $25 billion, against a figure FRI puts at $100 billion as of September 22, without naming the company.
Not every miss runs the same way. In a mid-2025 randomised trial run by Active Site, biosecurity experts predicted 22.5% of a sample of STEM undergraduates could complete three biorisk-relevant tasks with an LLM helping, virologists said 40% and superforecasters 16.2%. In the event, 5.2% completed all three.
FRI says plainly that its methodology "is biased toward identifying underestimates of AI progress rather than overestimates". A real value crossing a median forecast exposes a low call immediately, while a high call can only be judged once its resolution date arrives, and most questions have not resolved.
These are self-selected panels rather than a random sample, and the 24.6% and 9.7% cover those four benchmarks, not every question FRI has asked. The institute plans to keep publishing accuracy updates as LEAP questions resolve, wave by wave on the panel's report pages.
What that leaves is a scored record, which is more than most public AI predictions ever acquire, including Sam Altman's case against the people who called large language models a dead end. The last four years gave the field's own experts very little to be smug about.
