Task
Other
P
EleutherAI/pythia-70m
Other·EleutherAI· 579.5K· 83
gpt-neox
apache-2.0
95.6M params
dataset:EleutherAI/pilearxiv:2304.01373arxiv:2101.00027arxiv:2201.07311license:apache-2.0
The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact sam
# pull & run locally
pip install mlforge-sdk && mlforge pull EleutherAI/pythia-70m
pip install mlforge-sdk && mlforge pull EleutherAI/pythia-70m
Model details
Provider
EleutherAI
Framework
gpt-neox
Parameters
95.6M
Size
65 GB
License
apache-2.0
Downloads
579.5K
Likes
83
Paper
arXiv:2304.01373
Updated
2023-11-21
About EleutherAI/pythia-70m
The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same order. We also provide 154 intermediate checkpoints per model, hosted on Hugging Face as branches.