Mixed recovery provides twice as fast inference. Here’s how to implement this

Researchers at Kaist AI and MILA have introduced a new transformer architecture that is more memory and computationally efficient for large-scale language models (LLM). An architecture called mixed recovery (MOR) significantly improves the accuracy of the model and provides higher throughput compared to vanilla transformers.

Scaling challenges for LLMS

Today’s impressive capabilities of LLM are directly linked to an ever-growing number of sizes. However, as these models grow, memory footprints and computational requirements often become unacceptable, making both training and deployment difficult for organizations outside of hyperscale data centers. This led to searching for more efficient designs.

Efforts to improve LLM efficiency focus on two main ways: Parameter sharing and adaptive calculations. Parameter sharing techniques reduce the total number of unique parameters by reusing weights across different parts of the model, thus reducing overall computational complexity. For example, “layer tie” is a technique that reuses model weights across several layers. The adaptive calculation method is tailored to use only inference resources that the model needs. For example, “Early Termination” dynamically allocates computing by allowing the network to stop processing “simpler” tokens earlier.

However, creating an architecture that effectively integrates both parameter efficiency and adaptive computational calculations remains elusive.

How Mixing Recovery Works

Mixed recovery is a framework that combines parameter sharing and adaptive calculations to tackle the high computational demands of LLM. It is based on the concept of a recursive transformer. This is a model that applies a set of shared layers multiple times. Instead of a deep stack of unique layers, the recursive transformer divides the model into several “recursive blocks”, each with a shared pool of parameters. This design allows for more calculations without increasing the size of the model.

MOR enhances this recursive approach with two important components. The first is a lightweight router that intelligently assigns a specific recursive depth to each token. This concept is similar to the routing mechanism of the Mixture (MOE) model, in which routers direct tokens to specialized expert networks. However, in MOR, “experts” are different recursive depths, allowing the model to select the computational complexity to apply dynamically to each token. Determine how many times a layer’s shared block should be applied based on the complexity of the token or the required “depth of thought.” This only directs calculations when necessary most to avoid wasted cycles in parts that are easy to process input.

The second component is a more efficient key value (kV) cashing strategy. KV caching is a standard technique that speeds up generation by storing information from previous tokens, but it becomes a memory bottleneck in recursive models. MOR introduces a “recursive” KV caching mechanism that selectively stores and retrieves key value pairs only for still active tokens in a particular recursive step. This target cache reduces memory traffic and improves throughput without the need for complex and post-training changes.

As the researchers state in their paper, “Essentially, MOR allows models to efficiently adjust the depth of thinking for each token and unify parameter efficiency with adaptive calculations.”

MOR in operation

To test their framework, researchers trained MOR models ranging from 135 million to 1.7 billion parameters and compared them with vanilla and standard recursive baseline models for validation losses and a few accuracy benchmarks.

The results show great benefits. Given an equal training calculation budget, the MOR model achieved average less shot accuracy (43.1% vs. 42.3%) than the vanilla baseline, despite using nearly 50% of the parameters. When trained with the same amount of data, the MOR model reduced training time by 19% and reduced peak memory usage by 25% compared to the vanilla model.

The MOR architecture has also proven to be scalable. The performance of the vanilla model was slightly reduced at the smallest 135m parameter scale, but the gap closed rapidly as the model size increased. For models with parameters greater than 360m, the MOR matched or exceeded the performance of the standard transformer. Furthermore, MOR design dramatically increases the inference throughput. One MOR configuration achieved a 2.06x speedup at the vanilla baseline. For large-scale businesses, this can lead to significant operating costs reductions.

Paper co-author and Kaist doctoral student Sangmin Bae has destroyed the practical impact of emails to VentureBeat. “It’s difficult to provide accurate numbers, but reducing the model parameter size and KV cache footprint at a high level means that inference can be performed simultaneously on more samples,” he said. “This leads to an increase in the number of tokens processed at a time, making it possible to process long context windows.”

A practical path for enterprise adoption

The results of the paper come from a trained model from scratch, but the key question for businesses is how to adopt MOR without any large-scale upfront investments. According to BAE, the existing open source model is “an undoubtedly a more cost-effective approach.” He noted that while training the new model is simple, “the up-training approach could be more appropriate and efficient until the scalability of the MOR itself is fully verified.”

The adoption of MOR allows developers to introduce new architecture “knobs” to fine-tune the balance between performance and efficiency. This trade-off depends entirely on the needs of the application.

“For simpler tasks and scenarios, it may be beneficial to use models with recursive steps to increase flexibility and vice versa,” explained Bae. He emphasizes that “optimal settings depend heavily on a particular deployment setting,” encouraging teams to explore trade-offs based on findings in the paper.

From now on, the MOR framework is “modality-independent.” In other words, adaptive computational principles are not limited to text. This opens the door to significant efficiency improvements in the processing of video, audio, and other complex data types.

“We are extremely excited about the potential expansion into multimodality scenarios where efficiency is critical,” says Bae.

By dynamically adjusting the processing depth for each segment of the video or audio stream, MOR unlocked even greater cost savings and performance improvements, bringing the power of large-scale AI to a wider range of enterprise applications. As the paper concludes, MOR provides a “effective path to achieving large-scale model capabilities with significantly reduced computational and memory overhead.”

Daily insights into business use cases in VB every day

If you want to impress your boss, VB Daily has it covered. From regulatory shifts to actual deployments, it provides an internal scoop on what companies are doing with generated AI, allowing you to share the biggest ROI insights.

Please read our privacy policy

Thank you for subscribing. Check out this VB newsletter.

An error has occurred.