Task
Text Generation
P
ibm-research/PowerMoE-3b
Text Generation·ibm-research· 1.6M· 21
transformers
apache-2.0
3.4B params
arxiv:2408.13359license:apache-2.0region:us
Model Summary PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language
# pull & run locally
pip install mlforge-sdk && mlforge pull ibm-research/PowerMoE-3b
pip install mlforge-sdk && mlforge pull ibm-research/PowerMoE-3b
Model details
Provider
ibm-research
Framework
transformers
Parameters
3.4B
Size
19 GB
License
apache-2.0
Downloads
1.6M
Likes
21
Paper
arXiv:2408.13359
Updated
2024-09-24
About ibm-research/PowerMoE-3b
Model Summary PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language multi-choices, code generation, and math reasoning. Paper: https://arxiv.org/abs/2408.13359