Scaling Indic Pretraining to 2.8 Trillion Tokens With an Open Data Pipeline

Summary

NVIDIA describes an effort to scale pretraining for Indic-language AI to 2.8 trillion tokens. The article also presents an open data generation pipeline associated with that effort, outlining a way to produce training data at this scale. The available extract does not provide further details about the languages covered, the model architecture, evaluation results, or release terms.