Many engineering challenges end up with the same headaches. That means there are too many knobs to turn and too few opportunities to test them. Whether tuning the power grid or designing safer vehicles, each evaluation can be costly and involve hundreds of important variables.
Consider vehicle safety design. Engineers must integrate thousands of parts, and many design choices can affect how a vehicle behaves in a crash. Traditional optimization tools can have difficulty finding the best combination.
MIT researchers have developed a new approach that rethinks how a classic technique known as Bayesian optimization is used to solve problems involving hundreds of variables. When tested on realistic engineering-style benchmarks, such as power system optimization, this approach found optimal solutions 10 to 100 times faster than widely used methods.
Their method utilizes a foundational model trained on tabular data, automatically identifies the variables that are most important for improving performance, and iterates the process to hone in on a better solution. The underlying model is a huge artificial intelligence system trained on a vast set of common datasets. This allows it to be adapted to various applications.
Researchers’ tabular underlying models do not need to be constantly retrained as they work toward a solution, increasing the efficiency of the optimization process. This technology can be particularly useful in demanding applications such as materials development and drug discovery, as it provides significant speedups even for more complex problems.
“Modern AI and machine learning models have the potential to fundamentally change the way engineers and scientists create complex systems. We have come up with a single algorithm that can not only solve high-dimensional problems, but is also reusable, so it can be applied to many problems without having to start everything from scratch,” said Rosen Yu, a graduate student in computational science and engineering and lead author of a paper on the technique.
Yu is joined on the paper by Cyril Picard, a former MIT postdoc and research scientist, and Faez Ahmed, associate professor of mechanical engineering and core member of MIT’s Center for Computational Science and Engineering. This research will be presented at the International Conference on Learning Representations.
Improvements to proven methods
When scientists are trying to solve a multifaceted problem but have expensive ways to evaluate success, such as crash-testing cars to see how well each design performs, they often use a proven method called Bayesian optimization. This iterative method finds the best configuration for a complex system by building surrogate models that help estimate what to explore next while accounting for prediction uncertainty.
However, because the surrogate model must be retrained at each iteration, it can quickly become computationally difficult if the space of potential solutions is very large. Moreover, whenever scientists want to tackle a different scenario, they must build a new model from scratch.
To address both shortcomings, MIT researchers utilized a generative AI system known as Tabular Foundation Model as a surrogate model within a Bayesian optimization algorithm.
“Tabular foundational models are similar to ChatGPT for spreadsheets. The inputs and outputs of these models are tabular data, which is much more commonly displayed and used in the engineering realm than languages,” Yu says.
Similar to large-scale language models such as ChatGPT, Claude, and Gemini, this model is pre-trained on vast amounts of tabular data. This equips us to deal with a variety of prediction problems. Additionally, models can be deployed out-of-the-box without the need for retraining.
To make system optimization more accurate and efficient, the researchers employed tricks that allow the model to identify the features of the design space that have the greatest impact on the solution.
“A car may have 300 design criteria, but not all of them are the main drivers of the best design when trying to improve safety parameters. Our algorithm can smartly choose the most important features to focus on,” Yu says.
This is done by using a tabular underlying model to estimate which variables (or combinations of variables) have the most influence on the outcome.
Then, focus your search on the most influential variables instead of wasting time exploring everything equally. For example, if the size of the front crumple zone has increased significantly and the car’s safety rating has improved, that feature may have played a role in the enhancement.
Bigger problem, better solution
One of their biggest challenges was finding the best tabular underlying model for the task, Yu says. Next, we needed to connect it to a Bayesian optimization algorithm so that we could identify the most salient design features.
“Finding the most salient dimensions is a well-known problem in mathematics and computer science, but figuring out how to exploit the properties of the tabular underlying model was a big challenge,” Yu says.
After having the algorithmic framework in place, the researchers tested their method against five state-of-the-art optimization algorithms.
Across 60 benchmark problems, including realistic situations such as power grid design and car crash tests, their method consistently found the best solution 10 to 100 times faster than other algorithms.
“Our algorithm really shines when the dimensionality of optimization problems becomes larger,” Yu added.
However, their method did not outperform the baseline for all problems, including robotic path planning. This likely indicates that the scenario is not clearly defined in the model’s training data, Yu says.
In the future, the researchers hope to study ways to improve the performance of the underlying tabular model. They also want to apply their techniques to problems with thousands or even millions of dimensions, such as warship design.
“At a higher level, this work signals a broader shift: using fundamental models as algorithmic engines not only in perception and language, but also within scientific and engineering tools, allowing classical techniques such as Bayesian optimization to be extended into areas that were previously unrealistic,” Ahmed says.
“The approach presented in this study, which uses a pre-trained base model together with high-dimensional Bayesian optimization, is a creative and promising way to reduce the large data requirements of simulation-based design. Overall, this study is a practical and powerful step toward making advanced design optimization more accessible and easier to apply in real-world settings,” said Wei Chen, Wilson Cook Professor of Engineering Design and Chair of the Department of Mechanical Engineering at Northwestern University. Although he was not involved in this study.
