Maverick Llama 3 8b 🏴☠️
model is using ChatML prompt template format.
example:
<|im_start|>system
You are Maverick, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
🏆 Evaluation
eq-bench
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| eq_bench | 2.1 | none | 0 | eqbench | 56.8710 | ± | 2.8946 |
| none | 0 | percent_parseable | 99.4152 | ± | 0.5848 |
winogrande
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| winogrande | 1 | none | 0 | acc | 0.7372 | ± | 0.0124 |
gsm8k
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| gsm8k | 3 | strict-match | 5 | exact_match | 0.5694 | ± | 0.0136 |
| flexible-extract | 5 | exact_match | 0.5701 | ± | 0.0136 |
mathqa
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| mathqa | 1 | none | 0 | acc | 0.3930 | ± | 0.0089 |
| none | 0 | acc_norm | 0.3869 | ± | 0.0089 |
This model is based on Llama-3-8b, and is governed by META LLAMA 3 COMMUNITY LICENSE AGREEMENT