There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗? For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training. Here’s the work between downloading those and training a model 🧵
Marin 535B trained on 25T tokens from 152 datasets
For Marin's 535B run, the team built on 25 trillion tokens from 152 Hugging Face datasets with training-friendly licenses, sharing the pipeline work between downloading data and training.