Computer-aided design (CAD) is the go-to method for designing most of today’s physical products. Engineers can use CAD to convert 2D sketches into 3D models that can be tested and refined before sending the final version to the production line. However, this software is notoriously complex to learn, with thousands of commands to choose from. It takes a huge amount of time and practice to become truly proficient with the software.
MIT engineers aim to ease the CAD learning curve with AI models that use CAD software just like humans do. Given a 2D sketch of an object, the model quickly creates a 3D version with the click of a button or file option, much like an engineer would use software.
The MIT team has created a new dataset called VideoCAD. It contains over 41,000 examples of how 3D models are built in CAD software. By learning from these step-by-step videos of how different shapes and objects are constructed, the new AI system can now interact with CAD software just like a human user.
With VideoCAD, the team is building towards an AI-enabled “CAD copilot.” They believe such a tool could not only create a 3D version of a design, but also work with a human user to suggest next steps and automatically run build sequences that would be tedious and time-consuming to click through manually.
“AI has the opportunity to not only improve engineer productivity, but also make CAD more accessible to more people,” says Ghadi Nehme, a graduate student in MIT’s Department of Mechanical Engineering.
“This is important because it lowers the barrier to entry into design and allows people without years of CAD training to more easily create 3D models and unleash their creativity,” adds Faez Ahmed, associate professor of mechanical engineering at MIT.
Ahmed and Nehme, along with graduate student Brandon Mann and postdoctoral fellow Ferdous Alam, will present their findings at the Neural Information Processing Systems Conference (NeurIPS) in December.
click after click
The team’s new research expands on recent developments in AI-driven user interface (UI) agents. An AI agent is a tool that is trained to use software programs to automatically collect information online and perform tasks such as organizing it into an Excel spreadsheet. Ahmed’s group wondered if such a UI agent could be designed to use CAD. CAD includes more functionality and involves much more complex tasks than the average UI agent can handle.
In the new study, the team aimed to design an AI-driven UI agent that takes over the reins of a CAD program and creates a 3D version of a 2D sketch with each click. To do this, the team first looked at existing datasets of objects designed by humans in CAD. Each object in the dataset contains a set of high-level design commands such as Sketch Line, Circle, and Extrude that were used to construct the final object.
However, the team realized that these high-level commands were not enough to train the AI agent to actually use the CAD software. Real agents also need to understand the details behind each action. For example: Which sketch area should I select? When should I zoom in? And what part of the sketch should I extrude? To fill this gap, researchers have developed a system that translates high-level commands into user interface interactions.
“For example, let’s say you drew a sketch by drawing a line from point 1 to point 2,” says Nehme. “We converted these high-level actions into user interface actions: move from this pixel location, click, then move to the second pixel location, click with the “line” operation selected. ”
Ultimately, the team generated more than 41,000 videos of human-designed CAD objects. Each video describes a specific click, mouse drag, or other keyboard action performed by a human in real time. We then fed all this data into the model we developed to learn the relationships between UI actions and CAD object generation.
Once trained on this dataset, called VideoCAD, the new AI model will be able to take 2D sketches as input and directly control the CAD software by clicking, dragging, and selecting tools to build complete 3D shapes. The complexity of the objects ranged from simple brackets to more complex house designs. The team is training models on more complex geometries and envisions that CAD CoPilot could one day be made available to designers across a wide range of disciplines, both in models and datasets.
“VideoCAD is a valuable first step toward AI assistants that can help onboard new users and automate repetitive modeling tasks that follow familiar patterns,” said Mehdi Ataei, a senior researcher at Autodesk Research, which develops new design software tools and was not involved in the study. “This is an early foundation, and we look forward to successor products that span multiple CAD systems, richer operations such as assemblies and constraints, and more realistic and less complex human workflows.”
