Let’s say you’re reading a story or playing a game of chess. You may not have noticed, but at each stage of the road, your mind tracked how the situation (or “state of the world”) was changing. You can imagine this as a kind of event list. This is used to update your predictions of what will happen next.
Language models like ChatGPT track changes within your “mind” when you exit a code block or predict what you’ll write next. They usually make educated guesses using transformers (an internal architecture that helps models understand sequential data), but the system is sometimes incorrect due to flawed thinking patterns. Identifying and adjusting these underlying mechanisms makes the linguistic model more reliable prognostic addiction, particularly when using more dynamic tasks such as predicted weather and financial markets.
But are these AI systems the process of developing situations like we do? A new paper from researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and electrical engineering and computer science researchers show that the model instead uses clever mathematical shortcuts between each progressive step in order, ultimately making reasonable predictions. The team made this observation by moving under the hood of the language model and assessed how closely they could track objects that quickly change positions. Their findings show that engineers can control when using specific workarounds as a way to improve the system’s prediction capabilities.
Shell Game
Researchers analyzed the internal mechanics of these models using clever experiments reminiscent of classical intensity games. After the object was placed under the cup and shuffled in the same container, did you have to guess what the final location of the object is? The team used similar tests, and the model inferred the final arrangement of a particular number (also known as permutations). The model was given a starting sequence, such as “42135”, and was given instructions as to where and where to move each digit, such as moving “4” to the third position and then subsequent without knowing the final result.
In these experiments, we gradually learned that trans-based models predict the correct final arrangement. However, instead of shuffling numbers based on instructions given to them, the system aggregated the information between consecutive states (or individual steps in the sequence) and calculated the final permutation.
One go-to pattern observed by a team, known as the “associative algorithm,” essentially organizes nearby steps into groups and calculates the final guess. This process can be thought of as being structured like a tree with the initial numerical arrangement being “root”. As you move the tree up, adjacent steps are grouped into different branches and multiplied. At the top of the tree is the final combination of numbers, calculated by multiplying each result sequence on the branch.
The language model speculated that the final permutation was to use a crafty mechanism called “parity-related algorithms.” Determines whether the final arrangement is the result of equal or odd rearrangements of individual numbers. The mechanism then groups adjacent sequences from different steps before multiplying them, just like the associated algorithms.
“These behaviors show that transformers perform simulations through associative scans. Instead of changing state changes in stages, the models organize them into hierarchies.” “How do you encourage transformers to learn better state tracking? Instead of forming inferences about data in a human-like, continuous way, they should probably meet the approaches they naturally use when tracking state changes.”
“One of the research was to expand test time computing along the depth dimension, not the token dimension, by increasing the number of transformer layers, rather than the number of chain tokens considered during test time inference,” adds Li. “Our work suggests that this approach will allow transformers to build deeper reasoning trees.”
Through the visual glass
Li and her co-authors observed how association and parity-related algorithms work, using tools that can be peered into within the “mind” of language models.
They first used a method called “probing.” This shows what information flows through the AI system. Imagine examining the brain of a model and seeing the idea at a particular moment. In a similar way, this technique maps the system’s medium-term forecast for the final number arrangement.
Next, we used a tool called “activation patching” to show where the language model changes into situations. This includes injecting false information into certain parts of the network, interfering with some of the system’s “ideas” and keeping the others constant, and seeing how the system adjusts its forecasts.
These tools became apparent when algorithms generate errors and when the system “understands” how to correctly infer the final permutation. They observed that associative algorithms learn faster than parity-related algorithms and perform better even in long sequences. Li is overreliant on heuristics (or rules that allow for quick calculation of reasonable solutions) to predict permutations, resulting in more elaborate instructions for the latter difficulties.
“We found that when language models use heuristics early in their training, they start to incorporate these tricks into their mechanisms,” says Li. “However, these models tend to generalize worse than models that do not rely on heuristics. Since certain pre-training goals can block or encourage these patterns, we may consider designing techniques that discourage models from picking up bad habits in the future.”
Researchers note that their experiments were performed on small-scale language models that are fine-tuned with synthetic data, but found that model size had little effect on the results. This suggests that larger language models, such as GPT 4.1, are likely to produce similar results. The team plans to test language models of different sizes that have not been fine-tuned and examine hypotheses more closely by assessing their performance in dynamic real-world tasks such as code tracking and story evolution.
Postdoctoral Keon Vafa of Harvard University was not involved in the paper, but said the researchers’ findings could create opportunities to advance language models. “Many uses of large language models rely on tracking states, from providing recipes to writing code to tracking conversation details,” he says. “This paper makes great progress in understanding how language models perform these tasks. This progress provides interesting insights into what language models are doing and promises new strategies to improve them.”
Li wrote the paper with MIT undergraduate student Zifan “Carl” Guo and senior author Jacob Andreas. Their research was supported in part by open philanthropy, MIT Quest for Intelligence, the National Science Foundation, STEM’s Clare Booth Program for Women, and Sloan Research Fellowship.
The researchers presented their research this week at the International Conference on Machine Learning (ICML).
