Task
Text Generation
P
microsoft/Phi-tiny-MoE-instruct
Text Generation·microsoft· 831.7K· 38
transformers
mit
3.8B params
arxiv:2506.18349arxiv:2404.14219arxiv:2409.12136license:mitregion:us
Phi-tiny-MoE is a lightweight Mixture of Experts (MoE) model with 3.8B total parameters and 1.1B activated parameters. It is compressed and distilled from the base model shared by Phi-3.5-MoE and GRIN-MoE using the SlimMoE approach, then post-trained via supervised fine-tuning and direct preference optimization for instruction following and safety. The model is trained on Phi-3 synthetic data and
# pull & run locally
pip install mlforge-sdk && mlforge pull microsoft/Phi-tiny-MoE-instruct
pip install mlforge-sdk && mlforge pull microsoft/Phi-tiny-MoE-instruct
Model details
Provider
microsoft
Framework
transformers
Parameters
3.8B
Size
7.0 GB
License
mit
Downloads
831.7K
Likes
38
Paper
arXiv:2506.18349
Updated
2025-12-10
About microsoft/Phi-tiny-MoE-instruct
Phi-tiny-MoE is a lightweight Mixture of Experts (MoE) model with 3.8B total parameters and 1.1B activated parameters. It is compressed and distilled from the base model shared by Phi-3.5-MoE and GRIN-MoE using the SlimMoE approach, then post-trained via supervised fine-tuning and direct preference optimization for instruction following and safety. The model is trained on Phi-3 synthetic data and filtered public documents, with a focus on high-quality, reasoning-dense content. It is part of the SlimMoE series, which includes a larger variant, Phi-mini-MoE, with 7.6B total and 2.4B activated parameters.