Model Card for nano-phi-192M-v0.1
This is a continual effort from kenhktsui/nano-phi-115M-v0.1.
The model is not aligned.
Major differences:
- bigger tokenizer's vocab size
- addition of HuggingFaceTB/cosmopedia as training dataset
- training token: 19B vs 7B
How to use
To use the model, you will need transformer version >= 4.37.2
pip install transformers>=4.37.2
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("text-generation", model="kenhktsui/nano-phi-192M-v0.1")
pipe("I am a machine learning researcher. I work on", max_new_tokens=50, repetition_penalty=10.0)
Some metrics
- model
- hidden_size: 768
- num_key_value_heads: 8 (grouped query attention)
- num_attention_heads: 24
- num_hidden_layers: 6
- context length: 1024
- total params: 192M
- training:
- global steps: 36,000
Open LLM Leaderboard Evaluation Results
| Metric | kenhktsui/nano-phi-191M-v0.1 | kenhktsui/nano-phi-115M-v0.1 | microsoft/phi-2 (Reproduced) |
|---|---|---|---|
| Avg. | 29.24 | 28.68 | 61.53 |
| ARC (25-shot) | 24.15 | 21.93 | 61.52 |
| HellaSwag (10-shot) | 29.99 | 27.87 | 75.13 |
| MMLU (5-shot) | 25.46 | 25.30 | 58.23 |
| TruthfulQA (0-shot) | 44.30 | 46.01 | 44.46 |
| Winogrande (5-shot) | 51.54 | 50.99 | 74.51 |
| GSM8K (5-shot) | 0.0 | 0.0 | 55.34 |
Details:
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| arc_easy | 0 | acc | 0.4596 | ± | 0.0102 |
| acc_norm | 0.4070 | ± | 0.0101 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 25, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| arc_challenge | 0 | acc | 0.1911 | ± | 0.0115 |
| acc_norm | 0.2415 | ± | 0.0125 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 10, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| hellaswag | 0 | acc | 0.2833 | ± | 0.0045 |
| acc_norm | 0.2999 | ± | 0.0046 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| truthfulqa_mc | 1 | mc1 | 0.2583 | ± | 0.0153 |
| mc2 | 0.4430 | ± | 0.0152 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| hendrycksTest-abstract_algebra | 1 | acc | 0.2200 | ± | 0.0416 |
| acc_norm | 0.2200 | ± | 0.0416 | ||
| hendrycksTest-anatomy | 1 | acc | 0.2593 | ± | 0.0379 |
| acc_norm | 0.2593 | ± | 0.0379 | ||
| hendrycksTest-astronomy | 1 | acc | 0.1711 | ± | 0.0306 |
| acc_norm | 0.1711 | ± | 0.0306 | ||
| hendrycksTest-business_ethics | 1 | acc | 0.2400 | ± | 0.0429 |
| acc_norm | 0.2400 | ± | 0.0429 | ||
| hendrycksTest-clinical_knowledge | 1 | acc | 0.2566 | ± | 0.0269 |
| acc_norm | 0.2566 | ± | 0.0269 | ||
| hendrycksTest-college_biology | 1 | acc | 0.2639 | ± | 0.0369 |
| acc_norm | 0.2639 | ± | 0.0369 | ||
| hendrycksTest-college_chemistry | 1 | acc | 0.1800 | ± | 0.0386 |
| acc_norm | 0.1800 | ± | 0.0386 | ||
| hendrycksTest-college_computer_science | 1 | acc | 0.3300 | ± | 0.0473 |
| acc_norm | 0.3300 | ± | 0.0473 | ||
| hendrycksTest-college_mathematics | 1 | acc | 0.3000 | ± | 0.0461 |
| acc_norm | 0.3000 | ± | 0.0461 | ||
| hendrycksTest-college_medicine | 1 | acc | 0.2023 | ± | 0.0306 |
| acc_norm | 0.2023 | ± | 0.0306 | ||
| hendrycksTest-college_physics | 1 | acc | 0.2843 | ± | 0.0449 |
| acc_norm | 0.2843 | ± | 0.0449 | ||
| hendrycksTest-computer_security | 1 | acc | 0.2200 | ± | 0.0416 |
| acc_norm | 0.2200 | ± | 0.0416 | ||
| hendrycksTest-conceptual_physics | 1 | acc | 0.2511 | ± | 0.0283 |
| acc_norm | 0.2511 | ± | 0.0283 | ||
| hendrycksTest-econometrics | 1 | acc | 0.2807 | ± | 0.0423 |
| acc_norm | 0.2807 | ± | 0.0423 | ||
| hendrycksTest-electrical_engineering | 1 | acc | 0.2897 | ± | 0.0378 |
| acc_norm | 0.2897 | ± | 0.0378 | ||
| hendrycksTest-elementary_mathematics | 1 | acc | 0.2804 | ± | 0.0231 |
| acc_norm | 0.2804 | ± | 0.0231 | ||
| hendrycksTest-formal_logic | 1 | acc | 0.2143 | ± | 0.0367 |
| acc_norm | 0.2143 | ± | 0.0367 | ||
| hendrycksTest-global_facts | 1 | acc | 0.1700 | ± | 0.0378 |
| acc_norm | 0.1700 | ± | 0.0378 | ||
| hendrycksTest-high_school_biology | 1 | acc | 0.3226 | ± | 0.0266 |
| acc_norm | 0.3226 | ± | 0.0266 | ||
| hendrycksTest-high_school_chemistry | 1 | acc | 0.2759 | ± | 0.0314 |
| acc_norm | 0.2759 | ± | 0.0314 | ||
| hendrycksTest-high_school_computer_science | 1 | acc | 0.2700 | ± | 0.0446 |
| acc_norm | 0.2700 | ± | 0.0446 | ||
| hendrycksTest-high_school_european_history | 1 | acc | 0.2606 | ± | 0.0343 |
| acc_norm | 0.2606 | ± | 0.0343 | ||
| hendrycksTest-high_school_geography | 1 | acc | 0.3081 | ± | 0.0329 |
| acc_norm | 0.3081 | ± | 0.0329 | ||
| hendrycksTest-high_school_government_and_politics | 1 | acc | 0.3627 | ± | 0.0347 |
| acc_norm | 0.3627 | ± | 0.0347 | ||
| hendrycksTest-high_school_macroeconomics | 1 | acc | 0.2641 | ± | 0.0224 |
| acc_norm | 0.2641 | ± | 0.0224 | ||
| hendrycksTest-high_school_mathematics | 1 | acc | 0.2630 | ± | 0.0268 |
| acc_norm | 0.2630 | ± | 0.0268 | ||
| hendrycksTest-high_school_microeconomics | 1 | acc | 0.3403 | ± | 0.0308 |
| acc_norm | 0.3403 | ± | 0.0308 | ||
| hendrycksTest-high_school_physics | 1 | acc | 0.3113 | ± | 0.0378 |
| acc_norm | 0.3113 | ± | 0.0378 | ||
| hendrycksTest-high_school_psychology | 1 | acc | 0.2716 | ± | 0.0191 |
| acc_norm | 0.2716 | ± | 0.0191 | ||
| hendrycksTest-high_school_statistics | 1 | acc | 0.4491 | ± | 0.0339 |
| acc_norm | 0.4491 | ± | 0.0339 | ||
| hendrycksTest-high_school_us_history | 1 | acc | 0.2402 | ± | 0.0300 |
| acc_norm | 0.2402 | ± | 0.0300 | ||
| hendrycksTest-high_school_world_history | 1 | acc | 0.2363 | ± | 0.0277 |
| acc_norm | 0.2363 | ± | 0.0277 | ||
| hendrycksTest-human_aging | 1 | acc | 0.2197 | ± | 0.0278 |
| acc_norm | 0.2197 | ± | 0.0278 | ||
| hendrycksTest-human_sexuality | 1 | acc | 0.2824 | ± | 0.0395 |
| acc_norm | 0.2824 | ± | 0.0395 | ||
| hendrycksTest-international_law | 1 | acc | 0.2479 | ± | 0.0394 |
| acc_norm | 0.2479 | ± | 0.0394 | ||
| hendrycksTest-jurisprudence | 1 | acc | 0.2037 | ± | 0.0389 |
| acc_norm | 0.2037 | ± | 0.0389 | ||
| hendrycksTest-logical_fallacies | 1 | acc | 0.2393 | ± | 0.0335 |
| acc_norm | 0.2393 | ± | 0.0335 | ||
| hendrycksTest-machine_learning | 1 | acc | 0.1875 | ± | 0.0370 |
| acc_norm | 0.1875 | ± | 0.0370 | ||
| hendrycksTest-management | 1 | acc | 0.2039 | ± | 0.0399 |
| acc_norm | 0.2039 | ± | 0.0399 | ||
| hendrycksTest-marketing | 1 | acc | 0.1795 | ± | 0.0251 |
| acc_norm | 0.1795 | ± | 0.0251 | ||
| hendrycksTest-medical_genetics | 1 | acc | 0.3000 | ± | 0.0461 |
| acc_norm | 0.3000 | ± | 0.0461 | ||
| hendrycksTest-miscellaneous | 1 | acc | 0.2644 | ± | 0.0158 |
| acc_norm | 0.2644 | ± | 0.0158 | ||
| hendrycksTest-moral_disputes | 1 | acc | 0.2225 | ± | 0.0224 |
| acc_norm | 0.2225 | ± | 0.0224 | ||
| hendrycksTest-moral_scenarios | 1 | acc | 0.2726 | ± | 0.0149 |
| acc_norm | 0.2726 | ± | 0.0149 | ||
| hendrycksTest-nutrition | 1 | acc | 0.2353 | ± | 0.0243 |
| acc_norm | 0.2353 | ± | 0.0243 | ||
| hendrycksTest-philosophy | 1 | acc | 0.2283 | ± | 0.0238 |
| acc_norm | 0.2283 | ± | 0.0238 | ||
| hendrycksTest-prehistory | 1 | acc | 0.2099 | ± | 0.0227 |
| acc_norm | 0.2099 | ± | 0.0227 | ||
| hendrycksTest-professional_accounting | 1 | acc | 0.2411 | ± | 0.0255 |
| acc_norm | 0.2411 | ± | 0.0255 | ||
| hendrycksTest-professional_law | 1 | acc | 0.2458 | ± | 0.0110 |
| acc_norm | 0.2458 | ± | 0.0110 | ||
| hendrycksTest-professional_medicine | 1 | acc | 0.3897 | ± | 0.0296 |
| acc_norm | 0.3897 | ± | 0.0296 | ||
| hendrycksTest-professional_psychology | 1 | acc | 0.2141 | ± | 0.0166 |
| acc_norm | 0.2141 | ± | 0.0166 | ||
| hendrycksTest-public_relations | 1 | acc | 0.1818 | ± | 0.0369 |
| acc_norm | 0.1818 | ± | 0.0369 | ||
| hendrycksTest-security_studies | 1 | acc | 0.2490 | ± | 0.0277 |
| acc_norm | 0.2490 | ± | 0.0277 | ||
| hendrycksTest-sociology | 1 | acc | 0.2537 | ± | 0.0308 |
| acc_norm | 0.2537 | ± | 0.0308 | ||
| hendrycksTest-us_foreign_policy | 1 | acc | 0.2900 | ± | 0.0456 |
| acc_norm | 0.2900 | ± | 0.0456 | ||
| hendrycksTest-virology | 1 | acc | 0.1807 | ± | 0.0300 |
| acc_norm | 0.1807 | ± | 0.0300 | ||
| hendrycksTest-world_religions | 1 | acc | 0.1813 | ± | 0.0295 |
| acc_norm | 0.1813 | ± | 0.0295 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| winogrande | 0 | acc | 0.5154 | ± | 0.014 |
hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| gsm8k | 0 | acc | 0 | ± | 0 |