1. Model Introduction
JoyAI-LLM Flash-Base is a state-of-the-art mixture-of-experts (MoE) language model with 3 billion activated parameters and 48 billion total parameters. Trained with the Muon optimizer, JoyAI Flash-base achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities. JoyAI-LLM Flash series aim to accelarate high-throughput, latency-sensitive applications where cost per query must remain minimal.
Key Features
- Training-Inference Collaboration: apply Muon optimizer with dense MTP, develop novel optimization techniques to resolve instabilities while scaling up, delivering 1.3× to 1.7× the throughput of the non-MTP version.
- Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.
2. Model Summary
| Architecture | Mixture-of-Experts (MoE) |
| Total Parameters | 48B |
| Activated Parameters | 3B |
| Number of Layers (Dense layer included) | 40 |
| Number of Dense Layers | 1 |
| Attention Hidden Dimension | 2048 |
| MoE Hidden Dimension (per Expert) | 768 |
| Number of Attention Heads | 32 |
| Number of Experts | 256 |
| Selected Experts per Token | 8 |
| Number of Shared Experts | 1 |
| Vocabulary Size | 129K |
| Context Length | 128K |
| Attention Mechanism | MLA |
| Activation Function | SwiGLU |
3. Evaluation Results
4. License
Both the code repository and the model weights are released under the Modified MIT License.