Zephyr-Summarizer-3B
1. Introduction
Zephyr-Summarizer-3B produces faithful, abstractive summaries across documents, dialogues and transcripts.
2. Evaluation Results
Comprehensive Benchmark Results
| Benchmark | BriefLM | Zephyr-1B | Condenser | Zephyr-Summarizer-3B | |
|---|---|---|---|---|---|
| Core Reasoning Tasks | Math Reasoning | 0.613 | 0.638 | 0.643 | 0.660 |
| Logical Reasoning | 0.808 | 0.827 | 0.819 | 0.850 | |
| Common Sense | 0.770 | 0.774 | 0.744 | 0.796 | |
| Language Understanding | Reading Comprehension | 0.736 | 0.770 | 0.744 | 0.776 |
| Question Answering | 0.604 | 0.639 | 0.632 | 0.664 | |
| Text Classification | 0.806 | 0.797 | 0.793 | 0.850 | |
| Sentiment Analysis | 0.774 | 0.792 | 0.809 | 0.825 | |
| Generation Tasks | Code Generation | 0.706 | 0.726 | 0.744 | 0.752 |
| Creative Writing | 0.699 | 0.703 | 0.677 | 0.726 | |
| Dialogue Generation | 0.676 | 0.690 | 0.671 | 0.720 | |
| Summarization | 0.762 | 0.794 | 0.796 | 0.814 | |
| Specialized Capabilities | Translation | 0.798 | 0.803 | 0.797 | 0.831 |
| Knowledge Retrieval | 0.695 | 0.699 | 0.673 | 0.732 | |
| Instruction Following | 0.792 | 0.803 | 0.771 | 0.808 | |
| Safety Evaluation | 0.777 | 0.777 | 0.751 | 0.790 |
Overall Performance Summary
The Zephyr-Summarizer-3B demonstrates strong performance across all evaluated benchmark categories, with particularly notable results in reasoning and generation tasks.
3. Chat Website & API Platform
We offer a chat interface and API for you to interact with Zephyr-Summarizer-3B. Please check our official website for more details.
4. How to Run Locally
Please refer to our code repository for more information about running Zephyr-Summarizer-3B locally.
Temperature
We recommend setting the temperature parameter to 0.6.
5. License
This repository is released under the apache-2.0 license. The model supports commercial use.
6. Contact
If you have any questions, please contact us at info@zephyr.works.