Skip to content

The Actual News

Stay informed without the news wearing you out.

LLMs will provide accurate climate info, except when they won’t

LLMs will provide accurate climate info, except when they won’t

Summary

A University College London and UNESCO-led study tested 33 language models on climate science questions and policy scenarios. Models generally did well on factual questions, but struggled to assess evidence and predict how combinations or scale of policies would affect future warming.

Key Facts

  • The researchers tested 33 large language models, including frontier and open-weight models.
  • The study assessed models using 600 questions from several climate science benchmarks.
  • Flagship models answered about 78 percent of questions correctly on scientists’ confidence in conclusions from IPCC reports.
  • Frontier and large open-weight models generally scored around 90 percent on other factual benchmarks, while small open-weight models scored in the 70s.
  • On questions about whether evidence supported or refuted specific claims, the best models scored about 70 percent and the worst scored below 40 percent.
  • The researchers tested 656 questions about the effect of policies on global temperatures.
  • Models correctly predicted the direction of warming effects for single policies 72 to 89 percent of the time.
  • Models did no better than random guessing on questions about how pairs of policies would combine, and performed worse when asked to estimate the size of a policy’s effect.
Read the Full Article

This is a fact-based summary from The Actual News. Click below to read the complete story directly from the original source.

The week's biggest stories, one calm email every Monday.