Why AI models stumbled when a coal tax met a gas tax
Across 33 models, accuracy on pairs of climate policies averaged 33.8%. A simple strategy buried in the study’s appendix would have done better by giving the same answer every time.
A tax on coal can make gas more attractive to a power company. What happens if gas gets taxed too? In a UNESCO and University College London study, 33 AI models answered questions about pairs of climate policies with 33.8% accuracy on average. Choosing at random among the three possible answers would have scored 33.3%. The same models found it considerably easier to say which way a single policy would move global temperature.
The researchers could mark those answers without waiting until 2100. They ran each scenario through En-ROADS, a global simulator that connects energy demand, fuel choices, emissions and temperature. The test asked whether a language model could reproduce the simulator’s result from a written description. An answer counted as correct when the two agreed. It was never a prediction contest against a future thermometer.
The alternative fuel changes the answer
Consider the coal and gas example in the full report, section 2.4. Run the coal tax on its own, then the gas tax on its own, each time starting from the same baseline. Add the two temperature reductions on paper. Now run the simulator a third time with both taxes in force. In this example the combined reduction is larger than the sum.
One reason is substitution. The price of a fuel matters partly because of what a power company can buy instead. When only coal becomes dearer, gas remains an option. Put a tax on both fuels and the option that softened the effect of the first tax also becomes more expensive. Climate Interactive’s guide to the coal control explains how the change passes through competing energy sources and demand. The report offers substitution as a plausible explanation for this example. Other combinations can overlap and deliver a smaller reduction than the two isolated runs suggest.
Two scores from the same model
Across 656 questions, the researchers tested four abilities, from identifying the direction of a change to estimating its size and recognising interactions. Appendix A4.1 gives a useful side-by-side comparison. GPT-5.6 Sol matched the simulator on 90.6% of single-policy direction questions and 33.0% of interactions. Claude Opus 5 scored 87.5% and 34.4% respectively. A reader who saw only the first score for either model would have little reason to anticipate the second.
The distinction extends beyond climate science. In the IEA’s estimate of electricity demand at AI data centres, a building’s potential capacity and the load it actually draws are different quantities. A chatbot can explain that distinction in a sentence. Estimating what the grid will supply takes information about the buildings, their equipment and when it runs.
The climate experiment offers a practical way to interrogate a particularly fluent answer. Ask to see the baseline, each policy on its own and the two policies together, with the same assumptions in all four cases. Those four numbers let a reader check the claimed interaction. A paragraph about why coal gets more expensive, however persuasive, supplies only the beginning of that calculation.