NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation

Summary

NeMo Data Designer (NDD) is an open-source, general-purpose framework for generating multimodal synthetic data. Its declarative configuration format lets human or agent users define dataset columns covering text, code, structured outputs, images, embeddings, and statistical samplers that can steer diversity. A plugin system allows additional column types and capabilities to be added. Because the configuration is an inspectable artifact, workflows can be shared and reproduced. NDD builds preview and revision into the generation process: users can create a small sample, inspect the records, refine the specification, and rerun generation at full scale. At runtime, it resolves dependencies, schedules requests to user-provided model endpoints, and retries failed calls. The paper describes the framework's architecture and programming model and reports case studies involving structured, agentic, multimodal, and domain-specialized tasks, including datasets used for Nemotron model development and enterprise production deployments.