Back to News
Launches
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face Blog·October 16, 2026·1 min read
AI Summary
A new pretraining initiative has processed 2.8 trillion tokens of Indic language data, representing a significant scale-up in resources for South Asian language models. The project includes an open-source data generation pipeline to enable reproducible creation of large-scale multilingual datasets for Indian languages.
The full text couldn't be loaded here (the source may require a subscription).
View original at Hugging Face BlogKeep reading
Granite Switch 4.1: Chat, Decisions, and Speech from One EndpointHugging Face Blog · 15h agoWelcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from MetaHugging Face Blog · 1d agoWelcome FLUX.3 Image Klein in DiffusersHugging Face Blog · 1d agoHiRO-ACEv2: Simulating extreme weather at 3 km resolution for a fraction of the costHugging Face Blog · 1d ago
Was this useful?