Task
Text Generation
Z
Zyphra/Zyda
Text Generation · Zyphra· 9.5K
odc-by
3.0 TB
Zyda is a 1.3T language modeling dataset created by collecting open and high quality datasets and combining them and performing a uniform filtering and deduplication step. We find that Zyda performs extremely well in ablations and is at least comparable and potentially better to the best openly available datasets available, due to our meticulous post-processing pipeline. We think the best use of Z
# download instantly
mlforge datasets pull Zyphra/Zyda
mlforge datasets pull Zyphra/Zyda
Dataset details
Language
en
License
odc-by
Size
3.0 TB
Rows / images
4.4B
Creator
Zyphra
Downloads
9.5K
Source
huggingface_datasets
Updated
2024-06-19
About Zyphra/Zyda
Zyda is a 1.3T language modeling dataset created by collecting open and high quality datasets and combining them and performing a uniform filtering and deduplication step. We find that Zyda performs extremely well in ablations and is at least comparable and potentially better to the best openly available datasets available, due to our meticulous post-processing pipeline. We think the best use of Zyda is either as a standalone dataset for language model training up to the 1T scale, or in combination with Fineweb or Dolma for multi-trillion token training.