Researchers from MIT CSAIL, Google, and Northeastern University developed InstructMesh, a tool for refining AI-generated 3D designs through text prompts. By combining the TRELLIS 3D generator with GPT-4's reasoning, the system allows users to identify and fix structural flaws in generative blueprints before 3D printing physical objects.
Interactive 3D refinement
Generative 3D models often produce visually plausible shapes that fail to function in the real world, such as mugs that cannot hold liquid. InstructMesh addresses this by providing an interface where users can highlight specific sections of a mesh and request edits through natural language. The system bridges the gap between purely visual generation and functional manufacturing.
The architecture integrates Microsoft’s TRELLIS, which produces 3D structures from images or text, with the GPT-4 large language model. This combination allows the software to interpret user instructions for specific geometric modifications. The researchers demonstrated the tool by creating customized items including a functional multi-cup liquid dispenser and a denim-textured knee brace.
Application and current status
The project is currently at the research stage, demonstrated through various prototypes like a shrimp-shaped bristle bot that houses internal electronic components. While it simplifies the editing process for novices, the workflow still relies on the base capabilities of the underlying 3D generator and the reasoning accuracy of the language model.
The source describes successful fabrications of accessories and household items, but does not provide data on the software's performance with complex mechanical assemblies. The system is designed to handle mesh-based refinements rather than engineering-grade CAD constraints.
Original source
This report summarises the source below. Analysis is labelled separately; product and research claims remain attributed to their source.
Read the original at MIT