Technology news and discussion on TalkFuse.
LLMs will provide accurate climate info, except when they won’t
Researchers tested 33 large language models on climate questions and found that performance varies widely by task. The results suggest that models can often handle basic climate facts, but may give unreliable answers on policy impacts or when assessing evidence, which matters as more people use them for information.
Across 600 benchmark questions, leading models scored about 78 per cent on the most difficult questions about scientific confidence, while the smallest open-weight models scored about 47 per cent; results on other factual tasks were generally higher. On claims supported by evidence, the best models reached about 70 per cent accuracy and the weakest fell below 40 per cent. Models answered questions about the effect of individual policies correctly 72 to 89 per cent of the time, but questions combining policies were no better than random guessing, and estimates of how much warming policies would change were worse. The article text provided ends before reporting the findings from tests where users challenged models or identified themselves by role.
In short
- Models often answer basic climate questions well.
- Policy combinations and warming estimates are much less reliable.
- Larger models generally perform better, but still make errors.