FOLLOW US

A new peer-reviewed study reveals that AI models don't just produce answers — they secretly navigate invisible mathematical roadmaps embedded in their own thinking, and those roadmaps actually matter.

Samuel López · USA Herald - Here is a question that has kept AI researchers up at night for years: when an AI model works through a math problem step by step, is it actually thinking — or is it just making things up that sound smart? A new peer-reviewed study finally has a rigorous answer. And the answer is going to surprise people on both sides of the argument.

The short version: the thinking is real. The steps matter. And the AI models appear to know it — even when nobody told them.

Researchers from the Indian Statistical Institute in Kolkata and two French laboratories built a tool that does something no one had done at scale before. They took AI-generated math solutions and automatically extracted a hidden map from inside each one — a diagram showing exactly which calculation steps caused which other steps, all the way from the original question to the final answer. They called these maps Causal CoT Graphs, and they built 1,671 of them across three sets of math problems, from basic word problems all the way up to elite competition mathematics.

Then they ran an experiment that was almost cruel in its simplicity. They took fifteen different AI models — big ones, small ones, models from Google, Meta, DeepSeek, and others — and they surgically blocked each model's access to the key calculation steps identified in those maps. Not the whole reasoning trace. Just the specific nodes the map said were load-bearing. Then they watched what happened to the model's confidence in its final answer.

Every single model fell apart. Every time. Without exception.

The models didn't just get less confident. Their certainty in the answer collapsed so completely that the statistics were almost unprintable — in many cases the researchers simply wrote "p = 0" because standard notation couldn't capture how significant the result was. Across fifteen models, three datasets, and thousands of problems, blocking those specific reasoning steps destroyed the models' ability to answer. Which means those steps weren't decoration. They were load-bearing walls.

WHY THIS MATTERS TO YOU

Think of it like GPS navigation. For years, skeptics argued that AI reasoning steps were like a driver saying "I turned left, then right, then went straight" — a story invented after the fact to explain a route the GPS already knew. This study is the equivalent of proving that if you remove those turns from the map, the driver genuinely gets lost. The steps aren't a story. They're the route.

But the researchers found something even stranger in their second experiment. They compared how much probability each AI model assigned to traversing the "correct" causal path through its own reasoning — versus a random path of equal length through the same text. Consistently, across almost every model tested, the AI assigned higher probability to the mathematically valid path. The models appeared to internally "know" which steps in their own thinking were the important ones — even though no one had ever shown them the causal map.

The study also cracked open a debate about confidence versus creativity in AI. Some models, particularly those trained with reinforcement learning, became extremely confident in their reasoning — essentially locking in on one path and refusing to deviate. Others, particularly those built by distilling knowledge from larger AI teachers, stayed more uncertain and exploratory. When researchers measured which type performed better across multiple attempts at the same problem, the uncertain, exploratory models won. Their willingness to wander off the obvious path let them find solutions the overconfident models missed entirely.

The researchers are careful not to claim this settles every question about AI reasoning. Their mapping technique doesn't work well on geometry, abstract math, or common-sense problems. And proving the steps are causally necessary is not the same as proving AI thinks the way humans do. Those debates will continue.

What is no longer debatable is this: the intermediate steps AI models write down when solving math problems are doing real, measurable, causal work. The field's most prominent skeptics argued those steps were window dressing. This study, across fifteen models and nearly two thousand problems, says they were wrong.

THE BOTTOM LINE

The next time an AI walks you through its reasoning step by step, those steps aren't just for show. Scientists now have hard statistical evidence that the machine is genuinely using them to think — and that without them, it loses its way.

Source: Saha, Chaturvedi, Saha, Garain & Asher · "KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?" · ISI Kolkata, IRIT, LINAGORA Labs · Accepted Transactions of the Association for Computational Linguistics, January 2026 · arXiv:2507.11408v2

Dataset and code available open-source (MIT license) at espressovi.github.io/KisMATH

Follow USA Herald on X: @RealUSAHerald